Miro Dzakovic, Intel Corporation

MIDAS, the largest integrated database at Intel, manages Intel's internal manufacturing data from a variety of sources.  As Intel's manufacturing technologies keep changing, the MIDAS ETL applications must go through frequent upgrades.  When releasing these upgrades, the stakes are high: a single defect can spoil the data in the database, and the worst-case recovery from a backup would take days.  These challenges, which exist in many organizations that run on data, clearly call for fast and reliable regression testing. However, automated testing of database applications in industry is spotty.  No tool or technique is widely used.  This talk will highlight the approach behind 1TH, the regression testing tool that enabled weekly MIDAS upgrades while minimizing risks.  Apart from being a generic solution that is independent of database and ETL platforms, 1TH is particularly important for big data problems, such as entity resolution, and holds promise for applications that persist non-relational data.

We present that by collecting physical graph features of big discrete data in realtime, and feeding them to a classifier, we can leverage topology information to more effectively solve the universal fraud detection problem. We demo this technique by presenting a real customer's anti-fraud case--Combating Phone Scam On China Mobile Call Graph With Innovative Graph Features.

To elaborate, most existing telecom anti-scam systems leverage crowdsourcing. That is, mobile users proactively report spam numbers to a central service via mobile apps. The central service will maintain a blacklist and leverage it to alert potential victims in the future. However, in real-life, rule based anti-scam system is hardly effective due to the scarcity of the labeled data and the short-life of the labeled data. Scam phone numbers are highly dynamic, short-lived, and moving targets. As a result, anti-phone scam is a difficult imbalanced classification problem.

To effectively combat phone scam, we researched and discovered a highly effective process —via collecting innovative graph features on a mutable phone call graph in real-time, we can classify fradulent caller and alert victims in real-time. This process has been put in production to protect hundreds of millions of mobile users from phone scam since Dec 2016. To the best of our knowledge, such a system has neither been reported in academia nor in industry before. This is partly because there is no such large-scale, commercial grade mutable graph database to support this kind of attempt.

Specifically, we will cover
- The real-time anti-scam system architecture, which accepts real-time stream update on a 10B+ scale call graph, and computes and collects graph features in real-time on the graph.
- TigerGraph's innovative SQL-like query language, which is the enabler of the whole process
- Report a set of innovative graph features that is universal and effective in combating phone scam.

Yi Zhang, Renmin University of China; Xiaofeng Meng, Renmin University of China

In scientific domain, cutting edge creations and discoveries are often open accessible as texts in Web, papers and other carriers. Decentralized and unordered, they are hard to follow. By information extraction and reorganization, knowledge graph can help professionals follow latest and discover unknown scientific facts more effectively. Therefore, we will take microbiology for an example to show the process of large-scale scientific domain knowledge graph building by KGBuilder.
Taking texts as input, KGBuilder creates domain knowledge graph in 3 steps: identifying entities, relations and new links respectively by named entity recognition, relation extraction and linking prediction. In named entity recognition, there are some classic methods and tools in open domain, but usually perform poorly in specific one. Generally, relation extraction is completed by machine learning methods, requiring large amount of annotated data and manual feature engineering, which is expensive and time consuming. As for linking prediction, the unbalance between head entities and tail ones makes its data distribution unique.
Based on the above observation, KGBuilder can alleviate or solve corresponding problems. By combining BiLSTM, CRF and probabilistic methods, domain knowledge is added to named entity recognition in a simple and extensible way. Its relation extraction can automatically generate large amount of annotated data and extract features via distant supervision and neural network, reducing annotating cost. Inspired by Question Answering and rectilinear propagation of lights, KGBuilder puts forward TransMT to deal with the above unbalanced problem in link prediction, performing better than traditional models, especially translation ones.
By a preliminary attempt, KGBuilder draws a great blueprint for scientific facts following and discovering by knowledge graph, under which are machine learning and artificial intelligence.

Anurag Choudhary, Director of Engineering, heavily involved in development and delivery.

MapR implemented and runs the largest biometric database in the world - used for a wide variety of use cases - from entrance to exit out of India by citizens to health care fraud detection. This system has files and data on many of the majority of citizens in India for the purposes of supporting government functions. This system is live and fully operational - just as impressive the system has not had an outage in over 4 years.

Krzysztof Nienartowicz, University of Geneva

Gaia (http://sci.esa.int/gaia/28820-summary/) is a cornerstone ESA mission to create a 1.5 billion stars 3D catalog of our galaxy with unprecedented accuracy. Variability group hosted in Gaia Data Processing Centre in Geneva developed pipeline where the center of gravity is occupied by parallel DBMS: Postgres-XL. I will describe rationale, architecture and our involvement in development of Postgres-XL which enabled large scale processing and analytics for variability studies already publicly available at Gaia archive and to be expanded in following years.

WED 3: XProd: Efficient and Scalable Cartesian Product distribution for Map-Reduce primitives
Gopal Vijayaraghavan, Apache Tez; Zhiyuan Yang, Apache Tez; Hitesh Shah, Apache Tez

Cartesian product distributions are an unintended consequence of distributing non Equi-join algorithms which occur in geo-spatial and similarity problems. While there are existing methods to distribute such joins across a network using Map-Reduce primitives, they suffer from data duplication and network connection overheads which are avoidable. These overheads relegate these join queries into an undeserved reputation of being slow and hard to scale to billions of rows.

In this talk, we introduce a fast and scalable Cartesian distribution algorithm, built into Apache Tez, which allows for the execution of scalable non-equi joins within Apache Hive. The distribution algorithm uses the existing Map-Reduce primitives and only modifies the intermediate data interchange to achieve a full cross-product or any sub-set.

The Tez XProd Edge does not specify a join algorithm, allowing the use of Hive's distributed sorted merge or the distributed partitioned hash join algorithms, depending on the scale of operations. The physical distribution edge was designed to co-exist with a bloom semi-join edge, to reduce data-sets moving over the network based on one of the inputs. The XProd edge reduces data duplication by allowing one compressed partition of data to be routed to multiple destinations without modification by injecting itself as a routing table for the shuffle algorithm. The routing algorithms have multiple flavors which allows for the implementation to operate with fewer than O(N*M) network connections when input data is without skew, to group together partitions when skew is observed at runtime or distribute partitions in a skew-safe fashion when input skews are always expected.

We have implemented all these algorithms as part of Apache Tez and Apache Hive, with specific allowances for failure tolerance and to minimize total runtime of non-equi join queries, without any modification to the map-reduce assumptions within the tasks executing these join algorithms.

John Wu, Lawrence Berkeley National Laboratory

Many analysis operations, especially those from scientific applications, depend on data values that are hard to express in the popular data processing systems. Common operations, such as applying the Laplacian operator or computing vorticity from velocity, are typically represented as multi-ways joins that are very time-consuming to evaluate. In this work, we propose to capture the data dependency using a form of structural locality. We implement a User-Defined Function mechanism in a scientific data services framework to support operations with structural locality in data. This framework support data stored as multi-dimensional arrays, the most common form of data structure used in large-scale simulations. We implement this UDF mechanism, named ArrayUDF, as an in situ procedure to support these operation with structural locality. ArrayUDF allows users to define computations on adjacent array cells without the use of join operations, and executes the UDF directly on arrays stored in data files without loading their content into a data management system. Additionally, we present a thorough theoretical analysis of the data access cost to exploit the structural locality, which enables ArrayUDF to automatically select the best array partitioning strategy for a given UDF operation. In a series of performance tests on large scientific data sets, we have observed that ArrayUDF outperforms Apache Spark by as much as 2070X on the same high-performance computing system.

Kai Rothauge, ICSI and Dept. of Statistics, UC Berkeley

Apache Spark is a popular system aimed at the analysis of large data sets, but recent studies have shown that certain computations done in Spark are significantly slower than when using libraries written using a high-performance computing framework such as the Message-Passing Interface (MPI).

Ying Zhang, MonetDB Solutions; Richard Koopmanschap, MonetDB Solutions; Martin Kersten, MonetDB Solutions / Centrum Wiskunde & Informatica

Machine learning is quickly taking its well-deserved place in enriching Big Data applications through classification of facts, identifying trends, and proposing actions based on massive profile comparisons. Started out as a rather static workflow, first-generation machine learning has covered the following steps: 1) select salient features in the objects considered, 2) cleaning the collections to avoid statistical over/under fitting, and 3) a CPU intensive search to fit a model. A next challenge is to close this workflow loop and go after real-time learning and model adaptations.

This brings into focus a strong need to manage large collections of object features, avoiding their repetitive extraction from the raw objects, controlling the iterative learning process, and administering algorithmic transparency for post decision analysis. These are the realms in which database management systems excel. Their primary role in this world has so far mostly been limited to administration of the workflow states, e.g. administrating training sets used and conducting distributed processing.

However, should the two systems be considered separated entities, where application glue is needed to exchange information? How would such isolated systems be able to arrive at the real-time learning experience required? Inevitably, machine-learning applications will face the volume, velocity, variety and veracity challenges of big data applications.
Fortunately, the database community has been spending a tremendous effort on improving the analytical capabilities of database systems and improving the software eco-system at large. So when the well-established analytical database system MonetDB [1] met the popular new member of the machine- learning family TensorFlow, it was love at first sight. The click is immediate: they both can speak the same language (i.e. Python); they both have a strong focus on optimised processing on heterogeneous hardware (e.g. from multi-core in-memory processing to GPU/FPGA acceleration); they both serve the same communities (i.e. AI-enabled analytical applications); and most importantly this couple started out with a means of seamless communication (i.e. through the same binary data exchange format) [2].

In this short talk, I will first tell the story of how the recent in-database machine learning relationship between MonetDB (an open-source analytical columnar DBMS) and TensorFlow (an open-source machine learning library) came about. Then I will argue with an example application of entity linking using neural embeddings [3], that this relationship has all the potential to reach a happily-ever-after.

[1] P. A. Boncz, M. L. Kersten, and S. Manegold, Breaking the memory wall in MonetDB, Commun. ACM, 51(12): 77–85, 2008.
[2] M. Raasveldt and H. Mühleisen, Vectorized UDFs in Column-Stores, in Proceedings of the 28th International Conference on Scientific and Statistical Database Management, SSDBM ’16, pages 16:1–16:12, New York, NY, USA, 2016. ACM.
[3] T. Kilias, A. Löser, F. A. Gers, R. Koopmanschap, Y. Zhang, and M. L. Kersten, IDEL: In-Database Entity Linking with Neural Embeddings, submitted to VLDB 2018, https://github.com/tkilias/vldb2018-idel.

Swapnil Bhatia, Catalog Technologies Inc.

Catalog is building the world's first massive data storage platform using DNA as the storage medium. As an information storage medium, DNA is unique in its stability over millennia (fossils), information density (gigabits in cellular volume), and replicability (billions of copies, cheaply and efficiently). This makes it an attractive medium for high density storage of latency tolerant data. A central challenge in scaling DNA-based storage beyond narrowband rates is the cost-throughput trade off in classical letter-by-letter synthesis of DNA.

In this talk, I will present a glimpse of Catalog's technology for broadband DNA data storage and some of the unique challenges and opportunities presented by this new medium. These innovations result in dramatic cost reductions, which will enable DNA to become an economically attractive solution for high density latency tolerant applications.