Homework answers / question archive / Assignment Download some files (e

Assignment Download some files (e

Computer Science

Share With

---

Assignment

Download some files (e.g., 3 to 5 files) from Kickstarter datasets available under this link: https://webrobots.io/kickstarter-datasets

Merge the content of all days into a single file, which will be a big file. If the computer hard disk cannot handle that, then download it partially or keep it in a separate file, but the code should show that code is merging the files.

 

Write a code to search for a given keyword, e.g., “robot”, “fitness”, “wearable” and the result should present some columns including the name of the project, its category and funding amount. In other words, the user is reading from a file and presenting the output to the user in the java console (System.out.print...)

 

The search performed earlier now should include a memory.
By memory we mean that it keeps the track of what has been searched (inside a collection). In simple words, store inputs that we give to the search.
This means the search criteria and timestamp, should be stored in a collection (key and value).

 

There should be a method to call, and the result of this method prints the number of search terms, and frequency of search. For example, if I search for “wearable” two times, it prints: “wearable, 2 times”

 

No user interface is required to be developed, just java classes to handle everything is enough. However, respect the naming convention and java code structure.

 

The code should be readable, put enough comments, respecting java naming conventions, etc. and include a descriptive read.me file and implement a pom.xml when using external jar files.

 

1. Download the data from DBLP user: https://dblp.org/faq/1474679.html
   Since the datafile is too big, we can download a small part (>5MB data) of it and not all of it.

 

The following api can be used to extract the dblp dataset from the java code. https://dblp.org/search/publ/api?q=chat&format=json&h=1000

The query params can be modified to obtain different number of articles and search terms. 

 

• Create a Lucene index for each of its data, key word: “Conversational Agent”, “Chatbot”, “Personal Assistant”, “Smart Speaker”
• Query the dataset for each of these keywords “Conversational Agent”, “Chatbot”, “Personal Assistant” and “Smart Speaker”.
• The code should be readable, put enough comments, respecting java naming conventions, etc. and include a descriptive read.me file and implement a pom.xml when using external jar files.
• Plot the “response time” of the brute force search vs. Lucene ones for a keyword search, with different numbers of articles, including 1 article, 5 articles, 10 articles, 20 articles, 30 articles, ..., 100 articles. Something like the following Figure, one for each keyword

 

2. Use the above DBLP dataset used above and transfer the data in XML files into MonogoDB and Mysql.

XML file to be used can be provided over email

Write a query to count the number of given keywords per year, e.g., “gradient descent”, “wearable" keywords, for at least three years. The query should be implemented in both MySQL and MongoDB. Therefore, the project should have at least two separate classes, one for MySQL and one for MongoDB.

Write a range query that search for a given keyword in each from the start month to end month, like from Jan. 2018 to Mar 2019. The query should be implemented in both MySQL and MongoDB. Therefore, the code implementation should have at least two separate classes, one for MySQL and one for MongoDB.

The code should be readable, put enough comments, respecting java naming conventions, etc. and include a descriptive read.me file and implement a pom.xml file when using external jar files.