Topics in Data Science: Big Metadata

Data Science is Metadata Science

Data Science is one of those topics I can never get enough of…even the definition of Data Science as given by the the Data Science Association resonates with me; “the scientific study of the creation, validation and transformation of data to create meaning.

Data Science is a compelling topic because of the immense potential hidden in data sets, and unlocking that insight can successfully address our most significant societal challenges. The promise of Data Science is to more fully contribute to the greater good buy advancing our knowledge and leading to impactful discoveries. The impact of Big Metadata in the Data Science Framework is the topic I address in the following blog as an introductory topic in a new series of blogs called “Topics in Data Science”. Large data repositories generate massive amounts of metadata, enabling big data analytics to leverage technological and methodological advances in data science for the quantitative study of science. This blog post introduces a definition of Big Metadata in the context of data science and discusses the challenges and possibilities in Big Metadata analytics.

What is Metadata?

Metadata can more universally be thought of as value-added language that serves as an integrated layer in an information system. Metadata is structured data supporting functions associated with an object, an object being any “entity, form, or mode”. Metadata serves as connection to the lifecycle of the digital object being represented or tracks. While Big Data offers undreamed-of possibilities to find new data-driven solutions, Big Metadata can be perceived as data that encompasses information about the relationships among data, leading to the creation of a structure where data relationships can be explained.

Smart Data

Data Scientists work with large unstructured data sets, and these data sets are inherently messy, lacking the structures that make them suitable for analytics. To realize maximum value from a data lake, you must be able to ensure data quality and reliability, and make that data smart.  Metadata is inherently Smart Data because it provides context and meaning for data, and enables an action that draws on the metadata enhance connections that have been made. Smart Data is high quality, trusted data. It is accessible across the enterprise. Smart Data is actionable and can be ingested and understood by humans and/or machines.

Structure is everything

Data Science endeavors rely not only on data, but accurate description of the data - hence metadata.

In the practice of data science, much of the attention is focused on the beautiful visualizations or amazing discoveries made from analyzing large data sets. Little attention is given to the process the data scientist uses to get those results, and specifically the time-consuming process of preparing data. Data preparation accounts anywhere from 80–90% of the work of data scientists. They spend 60% of their time on cleaning and organizing data and 19% of their time on collecting data sets, meaning data scientists spend around a whopping 80% of their time preparing and cleaning their data for analysis. Detecting data anomalies and ameliorating data entry errors generally involves writing code, an intuitive part of the data exploration and confirmation process.

David Lyle, VP of Business Transformative Services at Informatica, wrote that “the difference between success and failure is proportional to the investment an organization makes in its metadata management system”.

Big Metadata Management

Large data sets used in data science have unique challenges pertaining to data management. Big metadata analytics requires careful design of datasets, paying attention to data structure. A misstep in data prep may cause a stalled server, never-ending loops, or very large datasets being exploded out of proportion or cases where data causes issues like disconnection from the network due to inactivity for a long time. There are also instances where, while merging datasets, data explodes out of proportion due to data matching issues. Successfully linking or merging data fields pivots on the extent to which metadata conforms to a standardized structure. Primarily, the largest advantage of this conceptual alignment of a standardized metadata structure on top of a metadata source is the creation of a formalized conceptual definition, allowing for a defined metadata interchange across datasets, fostering stable, reproducible results. Big metadata analytics performed on an ad hoc basis without standardized metadata exchange formats is more likely to suffer from mistakes made in both conceptual and computational workflows. Elevation of the metadata from the underlying storage allows its use in the meta-mining process.

---

• Funding
• AI
• Machine Learning
• artificial intelligence
• DARPA

---

DARPA Competency Aware Machine Learning Proposer's Day (CAML)

The goal of the DARPA CAML program is to building trusted Human-Machine Partnerships by developing the technology to allow autonomous systems to assess their own competence and communicate it to human teammates. I attended the proposers day, and the following article represents my notes and observations.

The opening session was presented by Dr Valerie Browning from the Defense Science Office at DARPA.. The Defense Science Office. The mission of the Defense Advanced Research Projects Agency (DARPA) Defense Sciences Office (DSO) is to identify and pursue high-risk, high-payoff research initiatives across a broad spectrum of science and engineering disciplines and to transform these initiatives into disruptive technologies for U.S. national security. Their investments represent a large fraction of DARPA’s current portfolio. Important Dates for DARPA CAML

• Abstract Due Date: March 6th 2019

• FAQ submission deadline: April 5, 2019

• Full proposal Due Date: April 22, 2019

DARPA CAML Strategy and Insight

The increasingly complex technologically sophisticated and dynamic military operational environment presents multiple fundamental challenges in how DARPA assists agencies in designing and planning for future military needs ranging from materials to platforms to systems, To address these challenges, a complex social behavior sciences and social systems thrust is a main alignment for DARPA DSO.

The CAML program is part of a broader agency wide activity to develop the next generation of AI technologies. As part of the DARPA AI NEXT campaign, DARPA continues its investment into fundamental AI technologies. DARPA has been investing in foundational AI technology since the inception of the AI field, and continues the investment into artificial intelligence. The DARPA AI next campaign, of which the Competency Aware Machine Learning program is a part, by investing in fundamental research in the science that underpins the technology.

Finding Funding within DARPA Competency Aware Machine Learning Proposer's Day (CAML)

Writing Your Abstract

DARPA operates on the principle that generating big rewards requires taking big risks. But how does the Agency determine what risks are worth taking? George H. Heilmeier, a former DARPA director (1975-1977), crafted a set of questions known as the "Heilmeier Catechism" to help Agency officials think through and evaluate proposed research programs. During the abstract phase of the proposal, the questions should be answered in an abbreviated format.

• What are you trying to do? Articulate your objectives using absolutely no jargon. 

• How is it done today, and what are the limits of current practice? 

• What is new in your approach and why do you think it will be successful? 

• Who cares? If you are successful, what difference will it make? 

• What are the risks? 

• How much will it cost? 

• How long will it take? 

• What are the mid-term and final “exams” to check for success?

Partner For Success

If your company wants to find funding with research projects, solid partnerships with companies who can prime the award is imperative. There is an art and a science to finding these relationships, and finding potential priming partners is much easier these days with business social networking like LinkedIn. In a future article I'll get into more detail around partnering for OTA’s vs Grants, and the different types of funding and how they drive a partnership.

Notes from the Contracting Officer

The content and quality of your abstract and proposal submission determines whether you will do business with DARPA. The scientific review panel evaluates each conforming proposal, and following the proposal instructions allows the program manager to understand fully your technical approach and how you will achieve the program milestones. A conforming proposal also allows an award to be made more quickly, and gives your proposal a better chance of success.

Keys to success for working with DARPA

• Read the entire BAA, and understand what objectives, milestones, and metrics the program manager is trying to achieve.

• Read the attachments included with the BAA, and follow the instructions closely.

• Submit all attachments and closely follow the proposal instructions; DARPA will evaluate only a conforming proposal.

Notes from the Program Director

A key ingredient in effective teams – whether athletic, business, or military – is trust, which is based in part on mutual understanding of team members’ competence to fulfill assigned roles. When it comes to forming effective teams of humans and autonomous systems, humans need timely and accurate insights about their machine partners’ skills, experience, and reliability to trust them in dynamic environments. At present, autonomous systems cannot provide real-time feedback when changing conditions such as weather or lighting cause their competency to fluctuate. The machines’ lack of awareness of their own competence and their inability to communicate it to their human partners reduces trust and undermines team effectiveness.

To help transform machines from simple tools to trusted partners, DARPA today announced the Competency-Aware Machine Learning (CAML) program. CAML aims to develop machine learning systems that continuously assess their own performance in time-critical, dynamic situations and communicate that information to human team-members in an easily understood format.

“If the machine can say, ‘I do well in these conditions, but I don’t have a lot of experience in those conditions,’ that will allow a better human-machine teaming,” said Jiangying Zhou, a program manager in DARPA’s Defense Sciences Office. “The partner then can make a more informed choice.” 

That dynamic would support a force-multiplying effect, since the human would know the capabilities of his or her machine partners at all times and could employ them efficiently and effectively.

In contrast, Zhou noted the challenge with state-of-the-art autonomous systems, which cannot assess or communicate their competence in rapidly changing situations.

“Under what conditions do you let the machine do its job? Under what conditions should you put supervision on it? Which assets, or combination of assets, are best for your task? These are the kinds of questions CAML systems would be able to answer,” she said.

Using a simplified example involving autonomous car technology, Zhou described how valuable CAML technology could be to a rider trying to decide which of two self-driving vehicles would be better suited for driving at night in the rain. The first vehicle might communicate that at night in the rain it knows if it is seeing a person or an inanimate object with 90 percent accuracy, and that it has completed the task more than 1,000 times. The second vehicle might communicate that it can distinguish between a person and an inanimate object at night in the rain with 99 percent accuracy, but has performed the task less than 100 times. Equipped with this information, the rider could make an informed decision about which vehicle to use.

DARPA has scheduled a pre-recorded webcast CAML Proposers Day for potential proposers on February 20, 2019. Details are available at: https://go.usa.gov/xE9aQ. 

The CAML program seeks expertise in machine learning, artificial intelligence, pattern recognition, knowledge representation and reasoning, autonomous system modeling, human-machine interface, and cognitive computing. To maximize the pool of innovative proposal concepts, DARPA strongly encourages participation by non-traditional proposers, including small businesses, academic and research institutions, and first-time Government contractors.