proper topic modeling, FINALLY!

So at 6AM this morning, I began following the instructions that Jon Goodwin kindly sent me for topic modeling with data from JSTOR data for researchers using his stuff plus some stuff written by Andrew Goldstone.

Subtract the 3 hours hours I spent dealing w/ life, and by noon I had, with huge huge huge support from both Jon and Andrew (like way more than you would expect from anyone who hadn’t actually been your BFF since grad school) topic modeled 1000 randomly sampled articles from Off OurBacks  (I really want the full 14000+ but must see if the fine folks @jstor will give them to me!)

Picture me people staring at my mac terminal yet not realizing I’m running linux rather than the MS-DOS of the ‘90.  Yes it took me 30 minutes to change a directory.  Then it turned out a line of code had to be changed.  Oh thank you @proghist2 for the preliminary lessons I did.  I opened file in komido, made change, and saved.  I was SO proud of myself.  Glitch again when it turned out I downloaded xml not csv files from JSTOR.  DOH, back again.  Another glitch when unzipped JSTOR file put things in subfolders, but I figured out on my own that I needed to move them.  Too bad I didn't remember the correct spacing for putting back together a line of code that was split by email.  At this point, Jon and Andrew had each responded to approximately 100000 emails from me, that I'm sure had them snorting coffee out their noses (as at one point I put in the $ that is used to denote a line of code to be entered at the command prompt.)  I got right down to the end and the final line of code would not run.  I erased the last file output, regenerated, and voila it did.

Trust me when I say, if I can do this, anyone can.  I am such a novice at doing anything without the aid of a GUI that newbie doesn’t even accurately capture my lack of knowledge. 

So JSTOR limits downloads without special permission to 1000 randomly sampled articles, and since I was eager to start I didn't fuss with parameters at all.   Still the results astounded me.  I absolutely did not expect clusters to appear so clearly.  I cannot wait to go back to JSTOR, to play with search terms and date limits to see if it remains as clear.

update here

0          5          women men sexual sex male woman rape girls female man pornography violence abuse prostitution gender sexuality rights harassment human violence against women

1          5          people time feel life children work things lot find make years change woman talk important part good thing jewish  maybe women's life cycle? 

2          5          lesbian lesbians feminist gay oob issue book read writing letter books dear paper radical reading article letters collective written   maybe women's writing, maybe lesbian feminism? 

3          5          women group movement political feminist conference issues feminists black groups feminism work con working discussion meeting white oppression present internal movement dynamics 

4          5          call send national center dc community write box university studies contact po program college black information june education info  maybe women and education?

5          5          court state law abortion rights case bill legal gay federal public child judge action supreme decision news order tion women and the law

6          5          health women woman medical found control study birth list drug cancer aids research breast pregnancy treatment doctors high clinic  women and health

7          5          women work workers union year money support pay care leave working jobs job labor national home members million welfare  women and work

8          5          back day time love house street days play night called made young good place home left head hear music

9          5          women police international united rights war violence world people peace government human groups country backs march held page report International women’s issues

I was curious about whether a sample, again of 1000, from a shorter time period would shift the results. 1970-1980 yielded

0          5          lesbian feminist lesbians issue gay oob movement feminists book conference issues political radical writing feminism read books dear letter  possibly lesbian feminism

1          5          court state abortion law rights bill case legal federal gay judge action supreme prison discrimination public decision york states women and the law

2          5          women groups national support government rights international united war union workers march police human members peace held group con International women’s issues

3          5          children child years family woman time mother home life day young mothers year age love back parents end told  possibly family life

4          5          people time work feel good things make group lot find part talk thought wanted thing place change movement felt

5          5          call dc write send box community show po contact june national information make st gay held art center area

6          5          women men sexual sex male violence rape woman pornography man human victims prostitution female sexuality power harassment death society violence against women

7          5          health women medical care hospital birth drug found woman control study cancer aids treatment percent high breast pregnancy doctors women and health

8          5          black white woman made jewish world life oppression days speak support struggle color racism page times country face stand  internal movement dynamics

9          5          women work working class issues social university education studies group center list discussion girls program ing public project world  women and work combined with women and education

five of the themes clearly carried over, indicated in large red above.  Two collapsed, women and work and women and education into one indicated in orange, and two new possibly themes emerged, indicated in blue.

ok it is late and I'm getting tired, but I just realized here is where the human interpretation of this process is faltering for me.

In run one at approx noon today I labeled this as

2          5          lesbian lesbians feminist gay oob issue book read writing letter books dear paper radical reading article letters collective written   maybe women's writing

at 4PM i labeled as possibly lesbian feminism

0          5          lesbian feminist lesbians issue gay oob movement feminists book conference issues political radical writing feminism read books dear letter  possibly lesbian feminism

sigh

I'm bleary eyed, but I can't resist one run at Feminist Studies  a more academically oritented publication than OOB with similar run 1972 v 1970 start but through to present

0          5          women family work labor social men workers children economic class working families domestic home gender male control household mothers work

1          5          women medical abortion health care law pregnancy child legal state rights welfare reproductive children policy pregnant social prison woman  health

2          5          sexual female woman male men sex life social human mother world love part sexuality nature fact sense desire experience  women’s sexuality

3          5          women black white american african history sexual south race racial native colonial political indian middle western university north cultural race/ethnicity

4          5          feminist gender women social feminism lesbian studies work political cultural identity university feminists theory politics history analysis critical power  academic feminism?

5          5          pp university paper feminist women studies ed american eds published mary history susan gender book judith social press essays  

6          5          art body work film disability images chicana beauty image female cultural artists people music woman space visual story bodies  images of women

7          5          women movement political feminist studies national issues research rights work groups international american program conference issue organizations public organization organzing

8          5          women de history female la woman mary early men marriage french century public life nineteenth political young american wrote

9          5          time back mother told people years life day make felt home knew feel long man good eyes left woman 

I need to look at this more, but I'm intrigued by not only the different themes such as sexuality race/ethnicity, but also by how the categories that overlap like work and health are constituted from different key words (academic activist divide?).  However its been twelve long hours of this work, so I think that is enough for now. 

Would love input into how I labeled and about emerging methodology