Thursday, February 22, 2024

Prompt Engineering

Prompt engineering involves designing and crafting prompts that effectively communicate the desired task or question to a language model like ChatGPT. 

The key components of prompt engineering include:

  1. Task Definition: Clearly defining the task or problem you want the language model to solve. This involves specifying the input format, expected output format, and any constraints or requirements.

  2. Context and Examples: Providing relevant context and examples to guide the language model's understanding of the task. This can include giving it sample inputs and corresponding outputs, demonstrating different cases or scenarios, and providing additional information or constraints.

  3. Prompt Structure: Designing the structure and format of the prompt to ensure clarity and consistency. This includes using appropriate language, specifying placeholders or variables, and organizing the prompt in a logical and coherent manner.

  4. Few-Shot Learning: Leveraging few-shot learning techniques to train the language model on a small number of examples. This helps the model generalize and adapt to new tasks or variations of existing tasks.

  5. Prompt Patterns: Utilizing prompt patterns or templates that capture common patterns or structures in prompt writing. These patterns provide a framework for constructing prompts and can help improve efficiency and effectiveness in generating desired outputs.

By focusing on these key components, prompt engineering contributes to improving prompt writing skills in several ways:

  • Precision: Prompt engineering helps in generating precise prompts that clearly communicate the desired task or question to the language model. This improves the accuracy and relevance of the model's responses.

  • Consistency: By designing consistent prompt structures and formats, prompt engineering ensures that the language model receives consistent inputs, making it easier to interpret and generate desired outputs.

  • Adaptability: Through few-shot learning, prompt engineering enables the language model to learn and generalize from a small number of examples. This enhances its ability to handle new tasks or variations of existing tasks.

  • Efficiency: Prompt patterns provide a systematic approach to prompt writing, saving time and effort by reusing proven structures and formats. This allows prompt engineers to focus on customizing prompts for specific tasks rather than starting from scratch.

  • Effectiveness: Well-engineered prompts improve the overall performance and reliability of the language model, leading to more accurate and useful responses. This enhances the user experience and the value derived from using the model.

By honing their prompt engineering skills, individuals can effectively harness the capabilities of language models and achieve better outcomes in various applications, such as natural language understanding, problem-solving, and content generation.

There are various types of prompt patterns that can be used to enhance prompt engineering with large language models like ChatGPT. Here are some examples:

  1. Input Prompt Patterns:

    • Asking for user input: Prompting the user to provide specific information or answer a question.
    • Providing alternatives: Offering multiple options for the user to choose from.
  2. Persona Prompt Patterns:

    • Adopting a persona: Writing prompts from the perspective of a specific character or persona.
    • Role-playing: Engaging in a conversation or interaction with the model as a specific persona.
  3. Instruction Prompt Patterns:

    • Asking for clarification: Requesting the model to provide more details or clarify a certain topic.
    • Asking for examples: Prompting the model to provide examples or demonstrate a concept.
  4. Formatting Prompt Patterns:

    • Specifying output format: Instructing the model to generate output in a specific format or structure.
    • Controlling verbosity: Guiding the model to be more concise or elaborate in its responses.
  5. Contextual Prompt Patterns:

    • Providing context: Including relevant background information or previous conversation history in the prompt.
    • Referring to previous responses: Referring to the model's previous answers or statements in the prompt.
  6. Goal-oriented Prompt Patterns:

    • Setting goals: Explicitly stating the desired outcome or objective in the prompt.
    • Requesting step-by-step instructions: Asking the model to provide a sequence of actions or steps to achieve a specific goal.

These are just a few examples of prompt patterns that can be used to structure prompts and guide the behavior of large language models. By leveraging these patterns effectively, users can achieve more accurate and desired responses from the models.


Chain-of-Thought Prompting Elicits Reasoning in Large Language Models


We explore how generating a chain of thought—a series of intermediate reasoning steps—significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain-of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting.



Refinement pattern 

This text outlines the concept of using the question refinement pattern to enhance interactions with large language models like ChatGPT. It suggests that by refining initial questions with the model's assistance, users can obtain more precise and contextually relevant queries. The process involves prompting the model to suggest improvements to questions and then deciding whether to use the refined version. The text emphasizes the importance of continuously striving for better questions to optimize interactions with the language model. Through an example involving a decision about attending Vanderbilt University, it illustrates how refining questions can lead to more informative and tailored inquiries. Additionally, it highlights how this pattern fosters reflection on the clarity and completeness of questions, helping users identify missing information and refine their queries accordingly. Overall, the text underscores the value of leveraging question refinement to generate better questions, enhance learning from model refinements, and address missing contextual elements for improved outputs.


Tuesday, June 27, 2023

prompts examples with ChatGPT / ChatGPT with RDBMS / Prompt Tuning

 Here are some examples of prompts you can use with ChatGPT:


General Information:

"Can you provide an overview of the history of artificial intelligence?"

"What are the benefits of exercise for overall health?"

"Explain the concept of blockchain technology."


Comparative Analysis:

"Compare the advantages and disadvantages of iOS and Android mobile operating systems."

"What are the differences between machine learning and deep learning?"

"Compare and contrast classical physics and quantum physics."


Problem-solving and Guidance:

"I'm trying to troubleshoot a computer network issue. Can you suggest some steps I can take to diagnose the problem?"

"How can I improve my time management skills and increase productivity?"

"What are some effective strategies for dealing with stress in the workplace?"


Exploratory Questions:

"What are some emerging trends in the field of renewable energy?"

"What are the potential impacts of artificial intelligence on job market in the next decade?"

"How is the COVID-19 pandemic affecting global supply chains?"


Scenario-based Questions:

"You are planning a trip to Paris. Can you recommend some must-visit attractions and local restaurants?"

"I want to start learning a new programming language. Which one would you suggest for beginners and why?"

"I'm considering pursuing a career in marketing. Can you provide insights into the key skills and qualifications needed in the industry?"

Remember to be as specific as possible in your prompts to get more targeted and accurate responses from ChatGPT. You can experiment with different prompt styles and variations to achieve the desired results.


---================================================================

An LLM (Language Model) can be used effectively with relational databases to perform a wide range of tasks, such as querying, data analysis, and generating SQL statements. Here's how you can utilize an LLM with a relational database:

  1. Query Generation: An LLM can assist in generating SQL queries for retrieving data from a relational database. You can provide the LLM with the desired criteria, filters, and conditions, and it can generate the corresponding SQL statement. This can be helpful when you're not familiar with the specific syntax or need assistance in constructing complex queries.


  2. Data Analysis and Exploration: Use an LLM to explore and analyze your database by asking questions or providing prompts. The LLM can provide insights and answers based on the data present in the database. For example, you can ask questions like "What are the top-selling products in the past month?" or "What is the average revenue per customer?" The LLM can generate SQL queries or analyze the data directly to provide the requested information.


  3. Schema Exploration and Documentation: An LLM can assist in understanding and documenting the database schema. You can ask the LLM questions like "What are the tables and columns in this database?" or "Describe the relationships between the tables." The LLM can provide information about the structure of the database, including table names, column names, data types, and relationships.


  4. Data Validation and Cleaning: Use an LLM to validate and clean your data by providing rules or conditions for data integrity checks. The LLM can generate SQL statements to identify and correct inconsistencies or errors in the data. For instance, you can ask the LLM to identify duplicate entries or missing values in a specific column.


  5. Natural Language Interface: An LLM can act as a natural language interface to your relational database. Instead of writing complex SQL queries, you can communicate with the database using plain English or natural language queries. The LLM can interpret the intent of your questions and generate the corresponding SQL statements for data retrieval or manipulation.


  6. Database Administration: An LLM can assist in managing and administering the database by providing information about database statistics, performance optimization techniques, or suggesting indexing strategies. You can ask questions about database administration tasks and receive recommendations based on best practices.

Remember that while an LLM can be a helpful tool for interacting with a relational database, it's essential to have a good understanding of database concepts, query optimization, and data integrity to ensure the accuracy and efficiency of your interactions.


---================================================================

Prompt tuning involves refining and optimizing prompts to improve the quality and relevance of the model's responses. Here are some techniques you can use for prompt tuning:

Be specific and explicit: Clearly specify the desired format, context, or type of response you want from the model. Provide detailed instructions and examples if necessary. The more specific and explicit your prompt, the better the chances of getting the desired output.

Control response length: If you want shorter or more concise answers, set a maximum token limit to restrict the response length. This can help avoid verbose or irrelevant outputs. Experiment with different token limits to find the right balance between length and completeness.

Experiment with temperature and top-k/top-p: Adjust the temperature parameter to control the randomness of the model's responses. Lower values like 0.2 make the model more focused and deterministic, while higher values like 0.8 introduce more randomness. Similarly, try different values for top-k (top-k sampling) or top-p (nucleus sampling) to influence the diversity and creativity of the responses.

Use system messages: System-level instructions can guide the model's behavior throughout the conversation. Use system messages to set context, remind the model of certain rules or constraints, or provide high-level instructions. These messages can help guide the conversation and shape the model's responses.

Iterate and refine: Prompt tuning is an iterative process. Experiment with different prompt variations, instructions, or techniques. Assess the model's responses and make adjustments based on the observed results. Continuously iterate and refine your prompts to improve the quality of the outputs.

Provide context and history: In a multi-turn conversation, include relevant context and history by prefixing previous messages. This helps the model maintain coherence and continuity in its responses. By referencing earlier parts of the conversation, you can guide the model to provide more consistent and contextually relevant answers.

Use human feedback: Solicit feedback from humans to evaluate the model's responses to different prompts. Collect feedback on the quality, relevance, and accuracy of the outputs. This feedback can guide you in further refining and optimizing your prompts to align with human expectations.

Fine-tuning: Consider fine-tuning the base model on a specific dataset or domain if you require more control over the outputs. Fine-tuning allows you to customize the model's behavior based on your specific needs and can lead to improved performance for specific tasks.

Remember that prompt tuning is a dynamic process, and the effectiveness of different techniques can vary depending on the specific task or domain. It requires experimentation, evaluation, and adaptation to optimize the prompts and achieve the desired outcomes.


Wednesday, June 21, 2023

Data Science - Feature Engineering

 Feature engineering employs various techniques to shape and enhance the features within a dataset. Some commonly used techniques include scaling, encoding, imputation, binning, and aggregation.

Scaling ensures variables are on a similar scale, while encoding transforms categorical variables into numerical form. Imputation techniques fill in missing values, and binning captures non-linear relationships. Aggregation involves deriving meaningful insights by aggregating data at a higher level.

Feature engineering steps: data understanding, feature selection, feature creation, and feature transformation. Each step contributes to refining the dataset and improving model performance.

Feature engineering is a dynamic and iterative process that requires continuous experimentation and evaluation. It allows data scientists to transform raw data into a feature-rich dataset that captures underlying patterns and relationships. However, it's important to strike a balance and avoid over-engineering, which can lead to overfitting or unnecessary complexity in models.
In conclusion, feature engineering is the cornerstone of successful data science and machine learning projects. 

Thursday, June 1, 2023

Data Lakehouse Architecture & Data Observability & Reverse ETL

https://drive.google.com/file/d/1ze7wNp91bhi4Pt40vsMPj9d2_27Q3VCr/view?usp=drive_link


https://docs.google.com/presentation/d/1v2mFcgCW15GH78ug_jHLPA7E120423EX/edit?usp=drive_link&ouid=106871833537371668656&rtpof=true&sd=true



https://drive.google.com/file/d/1cBG87MlaFacG6Fui0XyQf-F-tEPQpale/view?usp=drive_link




Snowflake Reverse ETL




Snowflake Reverse ETL



https://medium.com/snowflake/3-ways-data-engineers-at-snowflake-leverage-reverse-etl-1eb106bf6079


--------------------------


Reverse ETL on Snowflake | How to set up Hightouch on Snowflake for beginners


https://www.youtube.com/watch?v=KWNtUu9K_mk  


Adam Morton 


--------------------------


Hightouch Helps Companies Make Maximum Use Of Their Snowflake Data


https://www.youtube.com/watch?v=zZzO3WYLxOg



--------------------------

--------------------------


Sunday, January 9, 2022

Data Mesh Links

 https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017

https://venturebeat.com/2021/05/27/databricks-unifies-data-science-and-engineering-with-a-federated-data-mesh/

https://aws.amazon.com/blogs/industries/how-data-mesh-technology-can-help-transform-risk-finance-and-treasury-functions-in-banks/

https://aws.amazon.com/blogs/big-data/how-jpmorgan-chase-built-a-data-mesh-architecture-to-drive-significant-value-to-enhance-their-enterprise-data-platform/

Saturday, December 1, 2018

Decentralized storage options

Courtesy by --- 
Wulf Kaal, Entrepreneur, Technologist, Professor at University of St. Thomas School of Law (2011-present)

Currently decentralized blockchain applications have few options to store data. Decentralized storage options are:
  • Storing everything in blockchain itself
  • Peer to peer file system, such as IPFS
  • Decentralized cloud file storages, such as Storj, Sia, Ethereum Swarm, etc.
  • Distributed Databases, such as Apache Cassandra, Rethink DB, etc.
  • BigChainDB
  • Ties DB
Let’s consider them all in detail:
  1. Storing everything in blockchain itself: Storing everything in blockchain is the simplest solution. Currently most of the simple decentralized applications work exactly this way. However, this approach has significant drawbacks. First of all transactions to blockchain are slow to confirm. It may seem to be fast for money transfer (anyone can wait a minute), but it is extremely slow for a rich application data flow. Rich application may require many thousands transactions per second. Secondly, it is immutable. The immutability is the strength of blockchain that gives it high robustness but it is a weakness for a data storage. User may change their profile or replace their photo, still all the previous data will sit in blockchain forever and can be seen by anyone. The immutability results in one more drawback - the capacity. If all the applications would keep their data in blockchain, the blockchain size will grow rapidly, exceeding publicly available hard drive capacity. Full nodes can require special hardware. It may result in dangerous centralization of blockchain. That’s why storing data in blockchain only is not a good option for a rich decentralized application.
  2. Peer to peer file system, such as InterPlanetary File System. IPFS allows to share files on client computers and unites them in the global file system. The technology is based on BitTorrent protocol and Distributed Hash Table. There are several good moments. It is really peer to peer - to share anything first put it on your own computer. It will be downloaded only if anyone needs it. It is content addressable, so it is impossible to forge content by the given address. Popular files can be downloaded very quickly thanks to BitTorrent protocol. However it also has some drawbacks. You should stay online if you want to share your files. At least before someone becomes interested and wants to download them from you. It serves only static files, they can not be modified or removed once uploaded. And of course you can not search these files by their meaningful content.
  3. Decentralized cloud file storages: There are also decentralized cloud file storages that lift some of IPFS limitations. From the user’s point of view these storages are just cloud storages like Dropbox, for example. The difference is that the content is hosted on user’s computers who offer their hard drive space for rent, rather than in datacenters. There are plenty of such projects nowadays. For example, Sia, Storj, Ethereum Swarm. You don’t need to stay online to share your files anymore. Just upload the file and it is available in the cloud. These storages are highly reliable, fast enough, have enormous capacity. Still they serve static files only, no content search anyway and, since they are built on the rented hardware, they are not free.
  4. Distributed Databases: Since we need to store structured data and seek for advanced query capabilities we may look at the distributed noSql databases. Why noSql? Because strict transactional SQL databases can not be truly distributed due to the restrictions of the CAP-theorem. To make a database distributed we must sacrifice either consistency or availability. NoSQL databases choose availability over consistency replacing it with so called “eventual consistency” where all the database nodes in the network become consistent some time later. There are many mature realizations of such databases, for example MongoDB, Apache Cassandra, RethinkDB and so on. They are very good - fast, scalable, fault tolerant, support rich query language but still have fatal drawback for our application. They are not Byzantine-proof. All the nodes of the cluster fully trust each other. So any malicious node can destroy the whole database.
  5. BigChainDB: There is another project called BigChainDB that claims to solve the data storage and transaction speed problem. It is also a blockchain but with enormous data capacity and really fast transactions. Let us see how it is possible. BigChainDB is build upon RethinkDB cluster, I mentioned this NoSQL database on the previous slide. BigChainDB uses it to store all the blocks and transactions. That is why it shows such a high throughput - it is the one of the underlying noSQL database. All the BigChainDB nodes (denoted BDB on the slide) are connected to the cluster and have full write access to the database. Here comes a problem - the whole BigChainDB is not byzantine-proof! Any malicious BDB node can destroy the RethinkDB cluster. The BigChainDB team is aware of this problem and promises to solve it sometime in the future, however it is the corner stone of the architecture and changing it may not be possible.Anyway, BigChainDB may be good for a private blockchain. But in my opinion, to avoid confusion it should have been named BigPrivateBlockchain. It is not an option for a public storage.
  6. Ties DB: The currently available options could be a good public database. The closest to the ideal are the noSql databases. The only thing they lack is byzantine fault tolerance. The Ties.Network Database: ties.network is a deep modification of the Cassandra database and offers a preferable solution: The TiesDB inherits the majority of features from the underlying noSQL databases and adds byzantine fault tolerance and incentives. With these features it can become a public database and enable feature-rich applications on Ethereum and other blockchains with smart contracts. The database is writable by any user. But the users are identified by their public key and all the requests are signed. Once created, record remembers its creator who becomes an owner of the record. After that the record can be modified only by the owner. Everyone can read all records, because the database is public. All the permissions are checked on request and replication. Additional permissions can be managed via a smart contract.

Wednesday, November 28, 2018

What’s Your Data Strategy - 2 Types of Data Strategy

https://hbr.org/2017/05/whats-your-data-strategy?autocomplete=true


Saturday, July 28, 2018

Data Monetization.

https://www.accenture.com/us-en/insight-data-monetization-summary#block-datastrategy


Wednesday, December 20, 2017

Improving Data Quality with Machine Learning Techniques.



https://data.bloomberglp.com/promo/sites/12/750171296-FinDataGovernance.pdf

============================================================
https://www.bis.org/ifc/events/ifc_nbb_workshop/ifc_nbb_workshop_2d3.pdf

Improving Data Quality and Closing Data Gaps
with Machine Learning
Tobias Cagala*
May 5, 2017

=======================================================================


http://conteudo.icmc.usp.br/pessoas/gbatista/files/aai2003.pdf


An Analysis of Four Missing Data
Treatment Methods for Supervised Learning
Gustavo E. A. P. A. Batista and Maria Carolina Monard
University of S˜ao Paulo - USP
Institute of Mathematics and Computer Science - ICMC
Department of Computer Science and Statistics - SCE
Laboratory of Computational Intelligence - LABIC
P. O. Box 668, 13560-970 - S˜ao Carlos, SP, Brazil
{gbatista, mcmonard}@icmc.usp.br

==================================================================


http://dimacs-algorithmic-mdm.wdfiles.com/local--files/start/Methodologies%20for%20Data%20Quality%20Assessment%20and%20Improvement.pdf

Methodologies for Data Quality Assessment and Improvement
CARLO BATINI
Universit`a di Milano - Bicocca
CINZIA CAPPIELLO
Politecnico di Milano
CHIARA FRANCALANCI
Politecnico di Milano
and
ANDREA MAURINO
Universit`a di Milano - Bicocca


====================================================================

https://www.waterstechnology.com/reference-data-data-management/data-governance/3309956/webinar-making-enterprise-data-quality-a-reality

Webinar: Making Enterprise Data Quality a Reality


======================================================================

https://www.waterstechnology.com/innovationhub


=========================================================

https://www.bloomberg.com/professional/blog/machine-learning-plays-critical-role-improving-data-quality/

=============================================================


Monday, December 11, 2017

DATA QUALITY STRATEGY

DATA QUALITY STRATEGY: A STEP-BY-STEP APPROACH

“Strategy is a cluster of decisions centered on goals that determine what actions to take and how
to apply resources.”

Certainly a cluster of decisions – in this case concerning six specific factors
– will need to be made to effectively improve the data. Corporate goals will determine
how the data is used and the level of quality needed. Actions are the processes improved and invoked
to manage the data. Resources are the people,systems, financing, and data itself.
We can therefore apply the selected definition in the context of data,and arrive at the definition
of data quality strategy:

“A cluster of decisions centered on organizational data quality goals that determine the data
processes to improve, solutions to implement, and people to engage.”

EXECUTIVE SUMMARY
This paper will discuss:
• Goals that drive a data quality strategy
• Six factors that should be considered when building a strategy
• Decisions within each factor
• Actions stemming from those decisions
• Resources affected by the decisions and needed to support the actions.

You will see how these six factors — when added together in different combinations — provide the
answer as to how people, process and technology are the integral and fundamental elements of
information quality.

GOALS OF DATA QUALITY
Goals drive strategy. Data quality goals must support on-going functional operations, data management
processes, or other initiatives such as the implementation of a new data warehouse (DW), CRM
application, or loan processing system.

THE SIX FACTORS OF DATA QUALITY

When creating a data quality strategy there are six factors, or aspects of an organization’s operations that
must be considered. Those six factors include:
• Context — the type of data being cleansed and the purposes for which it is used
• Storage — where the data resides
• Data Flow — how the data enters and moves through the organization
• Work Flow — how work activities interact with and use the data
• Stewardship— people responsible for managing the data
• Continuous Monitoring — processes for regularly validating the data Figure 1 depicts the six factors centered on the goals of a data quality initiative, and shows that each factor requires decisions to be made, actions that need to be carried, and resources to be allocated.

TYING IT ALL TOGETHER

In order for any strategy framework to be useful and effective it must be scalable. The strategy framework provided here is scalable from a simple one-field update such as validating gender codes of male and female, to an enterprise-wide initiative where 97 ERP systems need to be cleansed and consolidated into 1 system. To ensure the success of the strategy, and hence the project, each of the six factors must be evaluated. The size (number of records/rows) and scope (number databases, tables, and columns) determines the depth to which each factor is evaluated.

Taken all together or in smaller groups, the six factors act as operands in data quality strategy formulas.

• Context by itself = The type of cleansing algorithms needed
• Context + Storage + Data Flow + Work Flow = The types of cleansing and monitoring technology implementations needed
• Stewardship + Work Flow = Near-term personnel impacts
• Stewardship + Work Flow + Continuous Monitoring = Long-term personnel impacts
• Data Flow + Work Flow + Continuous Monitoring = Changes to processes It is a result of using these formulas the people come to understand that information quality truly is the integration of people, process, and technology in the pursuit of deriving value from information assets. 



To help the practitioner employ the data quality strategy methodology,
the core practices have been extracted from the factors and listed here.
a) A statement of the goals driving the project
b) A list of data sets and elements that support the goal
c) A list of data types and categories to be cleansed(1)
d) A catalog, schema or map of where the data resides(2)
e) A discussion of cleansing solutions per category of data(3)
f) Dataflow diagrams of applicable existing dataflows
g) Work flow diagrams of applicable existing work flows
h) A plan for when and where the data is accessed for cleansing(4)
i) A discussion of how the dataflow will change after project implementation
j) A discussion of how the workfflow will change after project implementation
k) A list of stakeholders affected by the project
l) A plan for educating stakeholders as to the benefits of the project
m) A plan for training operators and users
n) A list of data quality measurements and metrics to monitor
o) A plan for when and where to monitor(5)
p) A plan for initial and then regularly scheduled cleansing

• Stewardship— people responsible for managing the data
• Continuous Monitoring — processes for regularly validating the data
Figure 1 depicts the six factors centered on the goals of a data quality initiative, and shows that each
factor requires decisions to be made, actions that need to be carried, and resources to be allocated.

Credit goes to Frank Dravis
https://pdfs.semanticscholar.org/5b5b/a15e8ea1bd89fe4d14d5e97ce456436291e0.pdf





Sunday, November 12, 2017

Functional Programming

https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/


https://www.youtube.com/watch?v=7Zlp9rKHGD4


https://pragprog.com/magazines/2013-01/functional-programming-basics

https://maryrosecook.com/blog/post/a-practical-introduction-to-functional-programming



===============================================================

Functional Programming based on lambda calculus.
It is without assignment statements.

statement has a side effect but where as expression has no side effects.

Expression based programming paradigm , using expressions as opposed to statements and combining expressions to form functions and combing functions to form complex behaviors.

=================================================================

Funtional Programming Jargon


Pure Functions


Immutability


Recursion


No Side Effects


Higher Order Functions


Category Theory


Lambda Calculus


Currying


Type Strictness : Int is type without classes but where string is a type with class.

Saturday, October 28, 2017

Best Content on Spark

https://spark-summit.org/2014/wp-content/uploads/2014/07/A-Deeper-Understanding-of-Spark-Internals-Aaron-Davidson.pdf

https://spark-summit.org/2014/wp-content/uploads/2015/03/SparkSummitEast2015-AdvDevOps-StudentSlides.pdf

Following is the Summary of the the above 2 pdf documents

Spark Major core components: 

– Execution Model
– The Shuffle
– Caching

Spark Execution Model 

1. Create DAG of RDDs to represent computation
2. Create logical execution plan for DAG
        Pipeline as much as possible
     • Split into “stages” based on need to reorganize data.
3. Schedule and execute individual tasks
        Split each stage into tasks
      • A task is data + computation
      • Execute all tasks within a stage before moving on
---------------------------------------------







“The main abstraction in Spark is that of a resilient distributed dataset (RDD), which represents a read-only collection of objects partitioned across a set of machines that can be rebuilt if a partition is lost.

Users can explicitly cache an RDD in memory across machines and reuse it in multiple MapReduce-like parallel operations.

RDDs achieve fault tolerance through a notion of lineage: if a partition of an RDD is lost, the RDD has enough information about how it was derived from other RDDs to be able to rebuild just that partition.”


“We present Resilient Distributed Datasets (RDDs), a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner.

RDDs are motivated by two types of applications that current computing frameworks handle inefficiently: iterative algorithms and interactive data mining tools.

In both cases, keeping data in memory can improve performance by an order of magnitude.”


An RDD can be created 2 ways: - 

1)  Parallelize a collection -
2)  Read data from an external source (S3, C*, HDFS, etc)

Life Cycle of a Spark Program

1) Create some input RDDs from external data or parallelize a collection in your driver program.

2) Lazily transform them to define new RDDs using transformations like filter() or map()

3) Ask Spark to cache() any intermediate RDDs that will need to be reused.

4) Launch actions such as count() and collect() to kick off a parallel computation, which is then optimized and executed by Spark.


Transformations (Lazy)

map()
intersection()
cartesion()
flatMap()
distinct()
pipe()
filter()
groupByKey()
coalesce()
mapPartitions()
reduceByKey()
repartition()
mapPartitionsWithIndex()
sortByKey()
partitionBy()
sample()
join() ...
union()
cogroup() ...

(lazy) - Most transformations are element-wise (they work on one element at a time), but this is not true for all transformations

Actions

reduce()
takeOrdered()
collect()
saveAsTextFile()
count()
saveAsSequenceFile()
first()
saveAsObjectFile()
take()
countByKey()
takeSample()
foreach()
saveToCassandra()

Types of RDDs

HadoopRDD • FilteredRDD • MappedRDD • PairRDD • ShuffledRDD • UnionRDD • PythonRDD • DoubleRDD • JdbcRDD • JsonRDD • SchemaRDD • VertexRDD • EdgeRDD


RDD Interface

1) Set of partitions (“splits”)
2) List of dependencies on parent RDDs
3) Function to compute a partition given parents
4) Optional preferred locations
5) Optional partitioning info for k/v RDDs (Partitioner)

Example: HADOOPRDD

Partitions = one per HDFS block
Dependencies = none
Compute (partition) = read corresponding block
preferredLocations (part) = HDFS block location
Partitioner = none

Example : FilteredRDD

Partitions = same as parent RDD
Dependencies = “one-to-one” on parent
Compute (partition) = compute parent and filter it
preferredLocations (part) = none (ask parent)
Partitioner = none


Example : JoinedRDD

Partitions = One per reduce task
Dependencies = “shuffle” on each parent
Compute (partition) = read and join shuffled data
preferredLocations (part) = none
Partitioner = HashPartitioner(numTasks)


Memory and Persistance

                                                         Worker Machine

Recommended to use at most only 75% of a machine’s memory
for Spark Minimum Executor heap size should be 8 GB Max
Executor heap size depends… maybe 40 GB (watch GC)
Memory usage is greatly affected by storage level and serialization format

- If RDD fits in memory, choose MEMORY_ONLY
- If not, use MEMORY_ONLY_SER w/ fast serialization library
- Don’t spill to disk unless functions that computed the datasets are very expensive or they filter a large amount of data. (recomputing may be as fast as reading from disk)
- Use replicated storage levels sparingly and only if you want fast fault recovery (maybe to serve requests from a web app)
- Intermediate data is automatically persisted during shuffle operations.

Spark uses Memory for 

RDD Storage: when you call .persist() or .cache(). Spark will limit the amount of memory used when caching to a certain fraction of the JVM’s overall heap, set by spark.storage.memoryFraction

Shuffle and aggregation buffers: When performing shuffle operations, Spark will create intermediate buffers for storing shuffle output data. These buffers are used to store intermediate results of aggregations in addition to buffering data that is going to be directly output as part of the shuffle.

User code: Spark executes arbitrary user code, so user functions can themselves require substantial memory. For instance, if a user application allocates large arrays or other objects, these will content for overall memory usage. User code has access to everything “left” in the JVM heap after the space for RDD storage and shuffle storage are allocated.


Serialization is used when:




Jobs       ---------------->          Stages  ---------->     Tasks




Schedulers




Lineage

“One of the challenges in providing RDDs as an abstraction is choosing a representation for them that can track lineage across a wide range of transformations.”

“The most interesting question in designing this interface is how to represent dependencies between RDDs.”

“We found it both sufficient and useful to classify dependencies into two types:
• narrow dependencies, where each partition of the parent RDD is used by at most one partition of the child RDD
• wide dependencies, where multiple child partitions may depend on it.”



Stages





Dependencies Narrow vs Wider

“This distinction is useful for two reasons:

1) Narrow dependencies allow for pipelined execution on one cluster node, which can compute all the parent partitions. For example, one can apply a map followed by a filter on an element-by-element basis. In contrast, wide dependencies require data from all parent partitions to be available and to be shuffled across the nodes using a MapReduce-like operation.

2) Recovery after a node failure is more efficient with a narrow dependency, as only the lost parent partitions need to be recomputed, and they can be recomputed in parallel on different nodes. In contrast, in a lineage graph with wide dependencies, a single failed node might cause the loss of some partition from all the ancestors of an RDD, requiring a complete re-execution.”



How do you know if a shuffle will be called on a Transformation?

-   repartition , join, cogroup, and any of the *By or *ByKey transformations can result in shuffles
-   If you declare a numPartitions parameter, it’ll probably shuffle
-   If a transformation constructs a shuffledRDD, it’ll probably shuffle
-   combineByKey calls a shuffle (so do other transformations like groupByKey, which actually
    end up calling combineByKey)

Common Performance issue checklist 

1. Ensure enough partitions for concurrency
2. Minimize memory consumption (esp. of sorting and large keys in groupBys)
3. Minimize amount of data shuffled 4. Know the standard library



Spark supports 2 types of shared variables: 

• Broadcast variables – allows your program to efficiently send a large, read-only value to all the worker nodes for use in one or more Spark operations. Like sending a large, read-only lookup table to all the nodes.

Broadcast variables let programmer keep a readonly variable cached on each machine rather than shipping a copy of it with tasks

For example, to give every node a copy of a large input dataset efficiently

Spark also attempts to distribute broadcast variables using efficient broadcast algorithms to reduce communication cost

• Accumulators – allows you to aggregate values from worker nodes back to the driver program. Can be used to count the # of errors seen in an RDD of lines spread across 100s of nodes. Only the driver can access the value of an accumulator, tasks cannot. For tasks, accumulators are write-only.


Accumulators are variables that can only be “added” to through an associative operation.
Used to implement counters and sums, efficiently in parallel.
Spark natively supports accumulators of numeric value types and standard mutable collections, and programmers can extend for new types.
Only the driver program can read an accumulator’s value, not the tasks

Spark usecases