Objectivity/DB Spark Adapter : Spark Adapter Tutorial
Spark Adapter Tutorial
The Spark Adapter tutorial will show how a Spark driver application interacts with an Objectivity/DB federated database, using Spark SQL and Scala.
ThingSpan Setup
If you haven’t already done so, install and configure ThingSpan:
1. Download the ThingSpan installer from the Objectivity Developer Network.
2. Run the installer to install ThingSpan with the Objectivity/DB Spark Adapter.
3. Set up the license file and environment variables as shown in the installed readme.html file.
Tutorial Setup
1. Download the tutorial sources from the Objectivity Developer Network Learning Center.
2. Extract the files, noting the top-level ObjySparkTutorial directory.
3. Follow the tutorial setup steps in the ObjySparkTutorial\readme file. These steps describe:
Where to find out about required third-party tools.
How to build the tutorial application.
How to create an Objectivity/DB federated database.
Note:You must complete the steps in the readme file before working through the various tutorial tasks.
The tutorial uses the Gradle build automation system to build the sample Spark driver application and to run various tasks. For more information about Gradle, refer to their website.
Tutorial Task Overview
The following table summarizes the tasks that you will complete as you work through this tutorial.
Tutorial Tasks
Store new objects in the federated database using a Spark SQL data frame.
Perform an inner join on a data frame loaded from the federated database and a data frame loaded from a JSON file, writing results back to the federated database.
Load an Objectivity/DB data frame in which objects of a given type are identified by their OIDs, then modify and write back particular objects.
Add new schema for objects that have relationships to each other, then learn how to distribute instances of these types using a round robin strategy across multiple storage locations.
Create objects and establish relationships between them using inner joins.