Scalytics Connect Release Notes

Scalytics Core

Apache Wayang is at the core of our products. Apache Wayang (incubating) is the only open-source, cross-platform data processing engine. Application developers specify applications using Apache Wayang’s API. 

‍

Scalytics Unique features:

• AI-based optimizer automatically picks an optimal configuration of class data processing frameworks, such as Java Streams or Apache Spark, to run applications on.
• Scalytics Core performs the program execution. It abstracts the different platform specific APIs and coordinates inter-platform communication.
• Applications can run on multiple data processing platforms, without changing the native code of the underlying platforms.
• Federated data processing: In-situ processing in different sites without moving raw data outside their origin.
• Build and execute cross platform machine learning pipelines in a unified way.
• NEW: Federated Machine Learning
  • Federated analytics by integrating multiple platforms across silos
  • Developers: Train ML models using federated learning in a platform agnostic way
• NEW: Supporting unsupervised learning (e.g., using K-means) and Stochastic Gradient Decent optimization technique for Federated Learning across supported data platforms
• NEW: Auditing compliance (who accessed what when) and training audits (basic)

‍

Data sources:

• PostgresSQL
• Columnar Data Files (e.g., CSV, Iceberg, Parquet, ORC)
• SQlite (e.g. Mobiles, Embedded)
• Local file systems
• Distributed file systems (e.g., HDFS, S3)
• Apache Kafka
• NEW: Remote files over http(s)
• NEW: JDBC based data sources

‍

Data Processing Platforms:

• Java 8 Streams
• Apache Spark / DataBricks
• Postgres
• SQLite
• Apache Flink / Confluent, Decodable
• NEW: Apache Kafka
• NEW: Tensorflow
• NEW: JDBC based platforms

‍

Programming APIs

• Java
• Scala
• Basic SQL
• New: Python (limited support)

‍

Runtime

• NEW: Actor-based runtime for building federated applications

‍

Scalytics Studio: Simplifying Machine Learning Workflow Design

Scalytics Studio is a cloud-native, low-code extension to Scalytics Core, designed to streamline Machine Learning workflow design and enhance data management.

With an intuitive graphical user interface, Scalytics Studio enables you to:

• Connect and Query Seamlessly: Effortlessly connect to various data sources and perform local queries.
• Unify and Join Data: Combine data from multiple sources with ease for comprehensive analysis.
• Transform Data Intuitively: Perform complex data transformations and execute them on the platform of your choice.

‍

Supported Data Sources:

• PostgresSQL
• Files on local or distributed filesystems (e.g., HDFS)
• NEW: Apache Kafka
• NEW: Files over http(s)

‍

Supported Platforms

• Java 8 Streams
• Apache Spark / DataBricks
• NEW: Tensorflow
• NEW: Apache Kafka

‍

Supported Data Transformations

• Map
• Filter 
• Reduce
• GroupBy
• Join
• Union
• Cross
• Train (for ML pipelines)
• Predict (for ML pipelines)