NLP

Install packages

# Setup - Run only once per Kernel App
%conda install openjdk -y

# install PySpark
%pip install pyspark==3.3.0

# install spark-nlp
%pip install spark-nlp==5.1.3

# restart kernel
from IPython.core.display import HTML
HTML("<script>Jupyter.notebook.kernel.restart()</script>")

Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 23.3.1
  latest version: 23.10.0

Please update conda by running

    $ conda update -n base -c defaults conda

Or to minimize the number of packages updated during conda update use

     conda install conda=23.10.0



# All requested packages already installed.


Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: pyspark==3.3.0 in /opt/conda/lib/python3.10/site-packages (3.3.0)
Requirement already satisfied: py4j==0.10.9.5 in /opt/conda/lib/python3.10/site-packages (from pyspark==3.3.0) (0.10.9.5)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: spark-nlp==5.1.3 in /opt/conda/lib/python3.10/site-packages (5.1.3)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
Note: you may need to restart the kernel to use updated packages.

Load required libraries

import json
import sparknlp
import numpy as np
import pandas as pd
from sparknlp.base import *
from pyspark.ml import Pipeline
from sparknlp.annotator import *
import pyspark.sql.functions as F
from pyspark.sql import SparkSession
from sparknlp.pretrained import PretrainedPipeline

Build Spark Session

spark = SparkSession.builder \
    .appName("Spark NLP")\
    .master("local[*]")\
    .config("spark.driver.memory","16G")\
    .config("spark.driver.maxResultSize", "0") \
    .config("spark.kryoserializer.buffer.max", "2000M")\
    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:5.1.3,org.apache.hadoop:hadoop-aws:3.2.2")\
    .config("fs.s3a.aws.credentials.provider","com.amazonaws.auth.ContainerCredentialsProvider")\
    .getOrCreate()

print(f"Spark version: {spark.version}")
print(f"sparknlp version: {sparknlp.version()}")

Spark version: 3.3.0
sparknlp version: 5.1.3

Read filtered data

%%time

import time
import sagemaker

bucket = "dsan-6000-group-35"
output_prefix_data_comments = "comments.parquet"
s3_path = f"s3a://{bucket}/{output_prefix_data_comments}"
print(f"reading comments from {s3_path}")
comments = spark.read.parquet(s3_path, header=True)

reading comments from s3a://dsan-6000-group-35/comments.parquet
CPU times: user 4.14 ms, sys: 15 µs, total: 4.16 ms
Wall time: 546 ms

print(f"shape of the comments dataframe is {comments.count():,}x{len(comments.columns)}")

[Stage 113:===================================================>   (14 + 1) / 15]

shape of the comments dataframe is 5,750,926x12

comments.printSchema()

root
 |-- year: integer (nullable = true)
 |-- month: integer (nullable = true)
 |-- subreddit: string (nullable = true)
 |-- id: string (nullable = true)
 |-- author: string (nullable = true)
 |-- link_id: string (nullable = true)
 |-- parent_id: string (nullable = true)
 |-- created_utc: timestamp (nullable = true)
 |-- body: string (nullable = true)
 |-- score: long (nullable = true)
 |-- gilded: long (nullable = true)
 |-- controversiality: long (nullable = true)

Data quality check

comments.show(5)

+----+-----+-----------+-------+--------------+---------+----------+-------------------+--------------------+-----+------+----------------+
|year|month|  subreddit|     id|        author|  link_id| parent_id|        created_utc|                body|score|gilded|controversiality|
+----+-----+-----------+-------+--------------+---------+----------+-------------------+--------------------+-----+------+----------------+
|2022|   10|TaylorSwift|irf7cfk|realscoutfinch|t3_xwui3g|t1_iracq55|2022-10-07 16:46:13|This this this! I...|    1|     0|               0|
|2022|   10|      Music|irf7cqw|    Alouitious|t3_xxz930|t1_irf6yfd|2022-10-07 16:46:17|4. Once you've bu...|    4|     0|               0|
|2022|   10|      Music|irf7dpm| ToastedSimian|t3_xy1kr4|t1_ireuzta|2022-10-07 16:46:29|Sadly, I'm a work...|    1|     0|               0|
|2022|   10|      Music|irf7drt|    tommykiddo|t3_xxz930|t1_irf39w8|2022-10-07 16:46:30|Ultimate Guitar h...|    3|     0|               0|
|2022|   10|      Music|irf7ghz|      monkee67|t3_xy0jgc| t3_xy0jgc|2022-10-07 16:47:03|MEH  \n\nBTW this...|   -2|     0|               1|
+----+-----+-----------+-------+--------------+---------+----------+-------------------+--------------------+-----+------+----------------+
only showing top 5 rows

NLP

# Filter comments to include only those mentioning Taylor Swift
swift_mentions_comments = comments.filter(comments.body.contains("Taylor Swift"))

swift_mentions_comments.show(5)

+----+-----+-----------+-------+--------------------+---------+----------+-------------------+--------------------+-----+------+----------------+
|year|month|  subreddit|     id|              author|  link_id| parent_id|        created_utc|                body|score|gilded|controversiality|
+----+-----+-----------+-------+--------------------+---------+----------+-------------------+--------------------+-----+------+----------------+
|2022|   11|TaylorSwift|ivp6h6i|      songacronymbot|t3_yqko6l|t1_ivp6fqj|2022-11-09 16:28:25|- SOTB could mean...|    2|     0|               0|
|2022|   11|TaylorSwift|ivp75f5|      songacronymbot|t3_yq56nj|t1_ivp749c|2022-11-09 16:32:57|- YOYOK could mea...|    1|     0|               0|
|2022|   11|TaylorSwift|iwh8avq|Impossible_Fudge_980|t3_yw2r6o| t3_yw2r6o|2022-11-15 16:57:57|1. Taylor Swift -...|   15|     0|               0|
|2022|   11|TaylorSwift|iwh8y98|   billsmafiamountup|t3_yw1pp8| t3_yw1pp8|2022-11-15 17:02:19|🤡 <— me think...|   41|     0|               0|
|2022|   11|TaylorSwift|iwh9ci9|    Madmaxinethemean|t3_yvjtpb|t1_iwh7nu5|2022-11-15 17:04:57|Edit: Image/Scree...|    2|     0|               0|
+----+-----+-----------+-------+--------------------+---------+----------+-------------------+--------------------+-----+------+----------------+
only showing top 5 rows

from pyspark.sql.functions import *

# generate two topics: 'music' and 'concerts'

swift_mentions_comments = swift_mentions_comments.withColumn(
    'music_related', col("body").rlike("(?i)music|album|song|track"))
swift_mentions_comments = swift_mentions_comments.withColumn(
    'concert_related', col("body").rlike("(?i)concert|tour|performance|stage"))

swift_mentions_comments.show(5)

+----+-----+-----------+-------+--------------------+---------+----------+-------------------+--------------------+-----+------+----------------+-------------+---------------+
|year|month|  subreddit|     id|              author|  link_id| parent_id|        created_utc|                body|score|gilded|controversiality|music_related|concert_related|
+----+-----+-----------+-------+--------------------+---------+----------+-------------------+--------------------+-----+------+----------------+-------------+---------------+
|2022|   11|TaylorSwift|ivp6h6i|      songacronymbot|t3_yqko6l|t1_ivp6fqj|2022-11-09 16:28:25|- SOTB could mean...|    2|     0|               0|         true|          false|
|2022|   11|TaylorSwift|ivp75f5|      songacronymbot|t3_yq56nj|t1_ivp749c|2022-11-09 16:32:57|- YOYOK could mea...|    1|     0|               0|         true|          false|
|2022|   11|TaylorSwift|iwh8avq|Impossible_Fudge_980|t3_yw2r6o| t3_yw2r6o|2022-11-15 16:57:57|1. Taylor Swift -...|   15|     0|               0|        false|          false|
|2022|   11|TaylorSwift|iwh8y98|   billsmafiamountup|t3_yw1pp8| t3_yw1pp8|2022-11-15 17:02:19|🤡 <— me think...|   41|     0|               0|        false|          false|
|2022|   11|TaylorSwift|iwh9ci9|    Madmaxinethemean|t3_yvjtpb|t1_iwh7nu5|2022-11-15 17:04:57|Edit: Image/Scree...|    2|     0|               0|        false|           true|
+----+-----+-----------+-------+--------------------+---------+----------+-------------------+--------------------+-----+------+----------------+-------------+---------------+
only showing top 5 rows

# Filter the DataFrame to get only the rows where 'music_related' is true
music_related_comments = swift_mentions_comments.filter(col("music_related"))

# Filter the DataFrame to get only the rows where 'concert_related' is true
concert_related_comments = swift_mentions_comments.filter(col("concert_related"))

music_related_comments.show(5)

+----+-----+-----------+-------+--------------+---------+----------+-------------------+--------------------+-----+------+----------------+-------------+---------------+
|year|month|  subreddit|     id|        author|  link_id| parent_id|        created_utc|                body|score|gilded|controversiality|music_related|concert_related|
+----+-----+-----------+-------+--------------+---------+----------+-------------------+--------------------+-----+------+----------------+-------------+---------------+
|2022|   11|TaylorSwift|ivp6h6i|songacronymbot|t3_yqko6l|t1_ivp6fqj|2022-11-09 16:28:25|- SOTB could mean...|    2|     0|               0|         true|          false|
|2022|   11|TaylorSwift|ivp75f5|songacronymbot|t3_yq56nj|t1_ivp749c|2022-11-09 16:32:57|- YOYOK could mea...|    1|     0|               0|         true|          false|
|2022|   11|TaylorSwift|iwh9g7y|        cookpa|t3_yvyclp|t1_iwh8yyq|2022-11-15 17:05:38|This, I don’t hat...|    0|     0|               1|         true|          false|
|2022|    8|TaylorSwift|imew9n5|songacronymbot|t3_x1mmuj|t1_imew68t|2022-08-30 17:42:33|- IKYWT could mea...|    1|     0|               0|         true|          false|
|2022|   12|TaylorSwift|j0m7i35| AimIsMyName01|t3_zo6s40| t3_zo6s40|2022-12-17 18:55:26|\-the 1 is extrem...|    4|     0|               0|         true|          false|
+----+-----+-----------+-------+--------------+---------+----------+-------------------+--------------------+-----+------+----------------+-------------+---------------+
only showing top 5 rows

# For music_related_comments
music_related_comments_count = music_related_comments.count()
music_related_comments_columns = len(music_related_comments.columns)
music_related_comments_shape = (music_related_comments_count, music_related_comments_columns)

# For concert_related_comments
concert_related_comments_count = concert_related_comments.count()
concert_related_comments_columns = len(concert_related_comments.columns)
concert_related_comments_shape = (concert_related_comments_count, concert_related_comments_columns)

print(f"Shape of music_related_comments is: {music_related_comments_shape}")
print(f"Shape of concert_related_comments is: {concert_related_comments_shape}")

[Stage 123:===================================================>   (14 + 1) / 15]

Shape of music_related_comments is: (25785, 14)
Shape of concert_related_comments is: (2513, 14)

from sparknlp.base import DocumentAssembler
from sparknlp.annotator import *
import nltk

nltk.download('stopwords')
from nltk.corpus import stopwords
nltk_stopwords = stopwords.words('english')

# Adjust the DocumentAssembler for comments
documentAssemblerComments = DocumentAssembler()\
    .setInputCol("body")\
    .setOutputCol("document")

tokenizer = Tokenizer()\
    .setInputCols(["document"])\
    .setOutputCol("token")

normalizer = Normalizer()\
    .setInputCols(["token"])\
    .setOutputCol("normalized")\
    .setLowercase(True)

lemmatizer = LemmatizerModel.pretrained() \
    .setInputCols(["normalized"]) \
    .setOutputCol("lemma")

stopwords_cleaner = StopWordsCleaner()\
    .setInputCols(["lemma"])\
    .setOutputCol("clean_tokens")\
    .setStopWords(nltk_stopwords)\
    .setCaseSensitive(False)

# Define the pipeline for comments
cleaning_pipeline_comments = Pipeline(stages=[
    documentAssemblerComments, 
    tokenizer, 
    normalizer, 
    lemmatizer, 
    stopwords_cleaner
])

# Fit the pipeline to the comments data
cleaning_model_comments_music_related = cleaning_pipeline_comments.fit(music_related_comments)
clean_swift_mentions_comments_music_related = cleaning_model_comments.transform(music_related_comments)

cleaning_model_comments_concert_related = cleaning_pipeline_comments.fit(concert_related_comments)
clean_swift_mentions_comments_concert_related = cleaning_model_comments.transform(concert_related_comments)

[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!

lemma_antbnc download started this may take some time.
Approximate size to download 907.6 KB
[OK!]

from pyspark.sql.functions import explode, col
import matplotlib.pyplot as plt
from wordcloud import WordCloud
# Extract words from the cleaned tokens
words = clean_swift_mentions_comments_music_related.select(explode(col("clean_tokens.result")).alias("word"))

# Aggregate word frequencies
word_frequencies = words.groupBy("word").count().orderBy(col("count").desc())

# Collect word frequencies to the driver
word_frequencies = word_frequencies.toPandas()

# Generate a word cloud
wordcloud = WordCloud(width=800, height=400, background_color ='white').generate_from_frequencies(dict(zip(word_frequencies.word, word_frequencies['count'])))

# Plot the WordCloud
plt.figure(figsize=(10, 5))
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.savefig('/root/fall-2023-reddit-project-team-35/data/plots/nlp/nlp_wordcloud_music_related_mx109.png', format='png', bbox_inches='tight')
plt.show()

from pyspark.sql.functions import explode, col
import matplotlib.pyplot as plt
from wordcloud import WordCloud
# Extract words from the cleaned tokens
words = clean_swift_mentions_comments_concert_related.select(explode(col("clean_tokens.result")).alias("word"))

# Aggregate word frequencies
word_frequencies = words.groupBy("word").count().orderBy(col("count").desc())

# Collect word frequencies to the driver
word_frequencies = word_frequencies.toPandas()

# Generate a word cloud
wordcloud = WordCloud(width=800, height=400, background_color ='white').generate_from_frequencies(dict(zip(word_frequencies.word, word_frequencies['count'])))

# Plot the WordCloud
plt.figure(figsize=(10, 5))
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.savefig('/root/fall-2023-reddit-project-team-35/data/plots/nlp/nlp_wordcloud_concert_related_mx109.png', format='png', bbox_inches='tight')
plt.show()

executive Summary

Our exploration of the language and specific vocabulary used in discussions of Taylor Swift reveals patterns in how fans and the public express appreciation and criticism of her work. Through NLP methods, we processed large amounts of text data and generated two comprehensive word clouds. These visual tools encapsulate words and phrases commonly used in Reddit comment sections when discussing Taylor Swift's music and live performances. This non-technical overview highlights universal terms and concepts associated with her, allowing us to understand her cultural influence.

analysis report

The goal of our NLP project is to uncover language trends in Taylor Swift’s discourse on social media. For Reddit comments, we implemented Spark NLP to systematically deconstruct comments mentioning Taylor Swift, using a series of processes including tokenization, normalization, lemmatization, and stopword cleaning. A data set was obtained that was ready for in-depth linguistic analysis.

We constructed two main word clouds. The first word cloud stems from a discussion focusing on Taylor Swift's musical output such as albums and songs, while the second focuses on her live performances. Notably, the word "album" appears frequently, indicating that her record collection generated significant engagement. Meanwhile, the words "tour" and "tickets" crop up frequently in concert-related discussions, highlighting the logistical aspects of her performances that elicit widespread fan engagement.

To ensure that the word cloud was not only informative but also visually compelling, we followed visualization best practices. Word clouds are presented with high contrast and clear boundaries for easy interpretation of results. These details ensure that our analysis is communicated effectively.

Our NLP-based analysis actually provides the basis for future in-depth sentiment analysis or topic trend analysis. As we parsed the data, we refined our goals, transitioning into focused examinations of specific topics that resonated most with our listeners and fans in our community. This methodological adaptability is a strategic decision in response to the evolving narratives in the data.