Scalytics Connect Release Notes

Scalytics Core

Apache Wayang is at the core of our products. Apache Wayang (incubating) is the only open-source, cross-platform data processing engine. Application developers specify applications using Apache Wayang’s API. 

‍

Unique features:

• AI-based optimizer automatically picks an optimal configuration of class data processing frameworks, such as Java Streams or Apache Spark, to run applications on.
• Scalytics Core performs the program execution. It abstracts the different platform specific APIs and coordinates inter-platform communication.
• Applications can run on multiple data processing platforms, without changing the native code of the underlying platforms.
• Federated data processing: In-situ processing in different sites without moving raw data outside their origin.

‍

Data sources:

• PostgresSQL
• Columnar Data Files (e.g., CSV, Iceberg, Parquet, ORC)
• SQlite (e.g. Mobiles, Embedded)
• Local file systems
• Distributed file systems (e.g., HDFS, S3)
• Apache Kafka

‍

Data Processing Platforms:

• Java 8 Streams
• Apache Spark / DataBricks
• Postgres
• SQLite
• Apache Flink / Confluent, Decodable
• Tensorflow
• Apache Giraph

‍

Programming APIs

• Java
• Scala
• Basic SQL
• Python

‍

Scalytics Studio

Scalytics Studio is a cloud-native, low code add-on to Scalytics Core for designing AI enhanced data management and ETL pipelines. 

Scalytics Studio delivers a graphical user interface that allows you to connect to and locally query different data sources or join data across multiple data sources in a very intuitive way. It also supports complex data transformations that can be processed on platforms of your choice.

‍

Supported Data Sources:

• PostgresSQL
• Files on local or distributed filesystems (e.g., HDFS)

‍

Supported Platforms

• Java 8 Streams
• Apache Spark / DataBricks

‍

Supported Data Transformations

• Map
• Filter 
• Reduce
• GroupBy
• Join
• Union
• Cross
• Train (for ML pipelines)
• Predict (for ML pipelines)