The project I worked on during the summer was to develop a generalized way to collect product information from mainstream fashion websites through anti-detective web scraping. To successfully finish the project, I needed to conduct thorough technical research on available scraping techniques, test methods to bypass website protocols, and implement strategic ways to store and update the scraped data in an efficient way so that the data can be fed for further use in scoring algorithms. After a lot of trials, I finally helped the startup formulate a generaled template to collect data with a standardized coding format. The newly added scrapers largely extended the number of records available in the database — they are scheduled weekly with the ability to update 400k new data records per run. This helps provide end users with more insights and transparency into non-sustainable fashion garments, empowering the startup to be able to acquire more users and later utilize those new records to train scoring models to provide more accurate scores for clothes. The work also fits the startup’s current business strategy — to expand data records and store more product information in their database.

With the extensive immersion in the whole process of data collection, transformation, and loading from local to the cloud data warehouse, I finally got a chance to see how data analytics and programming can be utilized in real-world data engineering. Starting from implementing Python packages to construct web scraper, manipulating text data to extract useful and useable information, to sorting and transforming unstructured data into a structured format, every step of my work is the real application and more advanced practice of what I learned in my graduate program. In addition to those standardized methodologies, the internship also provided me with more opportunities to contrast my in-school coursework with real-world situations,improving my ability to figure out a more industry-applicable method to tailor to a real business setting. Furthermore, I believe these experiences can also conversely contribute to my further study in the remaining semester, in that I can be more reflective in understading new knowledge and be able to connect the gained experience with upcoming new learnings.

The biggest challenge I faced was how to conduct proper anti-detection measurements to make sure the scraper could work healthily and efficiently. Since some of the websites have strict protocols to protect their internal data, it brings challenges for third-party users to bypass its detections. In order to overcome this challenge, I did a whole set of research on currently available detection techniques and resolutions for bypassing those detection protocols. For example, I set up rotating proxies and IP addresses to conceive users’ login information, switched user agents tactically to try to conceal the browsing history, and also played around between the direct request method and other frameworks or packages, say, using Selenium to mimic real human activities. Luckily, these techniques successfully help the scraper to work as expected, and the scraper smoothly collected information without complaining.

My takeaways are from my daily work and the interactions with people on the team. Through working on the data engineering project, my programming skills improved to a large extent. I gained first-hand experience in constructing databases, normalizing unstructured data, validating data structures, and building effective data scrapers from the ground up. These techniques really upskilled me, enhancing my overall understanding of traditional data engineering methods, which will be a great resource for me to deal with more complicated projects and programming challenges in the future. Furthermore, the necessary quality of being resilient and fearless about unforeseen challenges also equips me to be a better engineer. With my experience of building applications from scratch and overcoming challenges at the startup level, I do feel more confident in confronting future uncertainties. Furthermore, I must acknowledge that I learned so much by working with others on the team. Frequent interactions with the startup founder helped me gain first-hand experience in how to combine in-school knowledge with social ventures and the concept of impact entrepreneurship, as well as exposure to current technology products about sustainable fashion. By working with other data scientists and data engineers, I also learned how to tackle problems in a corporate environment and mitigate personal shortcomings through effective collaboration. I believe these lessons on interpersonal communication and interaction were also an important part of my summer journey, and those unforgettable memories and friendships will also bring lasting effects for my further life.