Details about the proposed projects
Students are free to select the topic and data selected for their challenge, but we
encourage students to select data in the catalan and/or spanish language.
In groups of two or three people, you will create a distributed system using Cloud
technologies with three main functionalities: (i) create a new text dataset extracting information from the
Web (Web crawler, Twitter APIs, …) and store it in Cloud Object Storage, and (ii) preprocess the text
dataset to build structured data (csv) that can be queried and analyzed later on. And (iii) create Python
notebooks to demonstrate date related queries, basic visualization, and sentiment analysis techniques over
the data.
To build the system, you will leverage the Lithops.cloud toolkit developed in
CloudButton. This toolkit enables to launch processes in the Cloud over Cloud Functions, and to store data
in Cloud Object Storage. We will provide training and examples of how to use this toolkit.
- Stage 1 - Data crawler: Massively parallel functions crawling data and storing it in Cloud Object
Storage. Obtain information from web pages or tweets and create a dataset of text data. Use FaaS backend
in Lithops to launch crawling process over serverless functions.
- Stage 2 - Data preprocessing: Produce structured data in csv format also stored in Cloud Object
Storage. As columns in the csv file we suggest date, geographic location, url, and sentiment analysis.
- Stage 3 - Python notebook: Demonstrate different queries in Python notebooks over your data. In
particular, apply sentiment analysis to data in different dates and periods and generate simple plots and
maps. Enable search queries over the data.
