Front-End Development 2: The Re-Front-Endening

Before starting my exploration of database options, I realized it would be a good idea to do a bit of informal testing with the current front-end/back-end logic/no data-storage build of Scaffold. I wanted to make sure that the UI would be usable enough that I wouldn't start working on integrating a database and then end up having to shoot back up the stack to fix the front-end.

After plugging in a few articles to , some necessary changes to my pre-testing UI have surfaced:

1) I'm realizing how helpful it would be to have a UI element that would directly allow for case-by-case feedback on the results. This would both be for making my testing process easier and for allowing users to provide more specific feedback down the road. I'm thinking a radio button next to each item in each category allowing for a tester (or feedback-inclined user) to indicate if a result was incorrectly categorized and why they considered it to be. Still trying to decide how to I would design this in a way that is easy to understand and also not a visual atrocity.

2) Speaking of visual atrocity, the current black and white list-format results are a little brain-numbing to look at for too long. To be clear, I don't mean to say that the problem is that they are bereft of artistic merit. The problem is that my eyeballs are physically in agony after using the app. The combination of eye-muscle strain occurring while scanning for the location of a specific item's usage in an article and overall blinding effect of the white background is truly exceptional for conjuring ocular pain. While I could leave it as is and appreciate that Scaffold will be a stand-out application in at least one way, I'm not so keen on beginning all of my demos with the phrase, "now this may hurt a little..."*

 3) Scaffold's results and line-numbered article need to wrap to fit the browser window. Currently, they don't do this and it's a guaranteed bad time since Scaffold is almost always going to be occupying one part of the screen while the user has a document editor/browser/etc. open at the same time. In my case, I need to be able to stare at Scaffold (in app form) while also staring Scaffold (in code form).

That's all of the main UI fixes I have decided need to be dealt with before I move on. Let the HTML encore begin!

*EDIT- I realized going back to this that I never actually specified what my next work item was to fix the eye-hurt interface. I'm going to try out a couple of alternative color schemes to see which one has good contrast but isn't over the top. I also intend to break up the text blocks into more readable tables or at least a more eye-scan-friendly arrangement of text

Working With Flask Has Been Pretty Groovy So Far

The incredibly basic front-end of my dreams has been integrated with my back-end logic. I might remember to post screenshots or a bitty demo vid soon. Actually, I definitely want to do that, please don't let me forget to! Also, I have cleaned up an outstanding issues on the back-end where nltk did not appear able to recognize non-ascii value representations of unicode general punctuation characters.

 Right now, I have solved this by checking all of the characters in the input and replacing all characters with unicode values higher than 127 with an equivalent sub 127 character or ignoring the character if no replacement has been given. It's not a particularly eloquent solution, but it has been working for the few test documents I have used. For now, I would prefer to move onto creating the back-end database, nailing down a concrete testing process, establishing testing benchmarks, and getting started with more serious testing. Then I can worry about which parts of the code are most in need of revision and what's worth getting done before the first release.

For now, time to get down with trying to integrate a MySQL database (or similar) with the app.

Miscellaneous Personal Notes to Consider as Scaffold Moves Forward

I have some other thoughts that I wanna throw out into the ether without cluttering that last blog post I just made that was already getting pretty long.

First, I have been working through the Stanford machine learning course on Coursera and it has been an awesome time so far! I am excited to finally be following a structured machine learning curriculum and am looking forward to being able to build a much smarter v2 of Scaffold in a couple of months. Also, if it seems like my work on Scaffold is slowing down a bit, it is most likely because I am now working through the ML course material (as well as job hunting/interview studying/keeping my cpp fresh).

Secondly, I just really need to throw out into the ether that I am not particularly fond of front-end engineering and I would be lying if I said that I am looking forward to the UI implementation/design portion of this project.

I appreciate the field of design and I don't find it insane that there are developers who find front-end fun. I have minors in psychology and studio art which, along with the CS major, seems like the recipe for a front-end developer. Instead, it has been the recipe for a back-end developer who is interested in using computation to model intelligence and making surrealist sketches of cats officiating at bird weddings as a side hobby.

I am committed to my vision of Scaffold taking form as a web app. I want it to be a tool accessible that non-technical individuals can benefit from in addition to technical users. I can live through a bit of HTML boredom to make that happen. In addition, I have never constructed a web app top-to-bottom on my own and it's something that I stand to learn a lot from doing. This project is also largely fueled by my desire to strengthen and extend my skills as a developer.

Worst-case scenario I will gain experience in an area of the stack I don't work much in. Best-case scenario, I have a grand time, take back everything I said before, and broaden my horizons. My current dislike of front-end engineering doesn't make me more useful, more interesting, or a better back-end engineer. But I wanted to own up to it here because it honestly has effected how I have gone about the development process so far (i.e. building an inadvisable amount of back-end functionality before considering how it was going to integrate into the rest of a web app stack).

I want to think of clever way to wrap this up but honestly I have had to pee for like forty-five minutes so I'm just going to post this so I can close my laptop and run to the restroom.

Laying Out The Plan For This Whole "Building a Web App" Thing

So, I am returning to my original plan and not continuing to implement the functionality to extract and parse text content from user-given urls. The reasons for this being:

1) Having my server held responsible for opening and loading the contents of every user submitted url is probably not a great idea

2) If I want to make Scaffold into a web app, I need to divert from rewriting and adding to back-end functionality. Instead, I should ensure that the functionality I have already implemented is being constructed in a way that will work well with the web app architecture. Particularly when considering additional resources that might be necessary to improve Scaffold's overall identification error rate.

As it stands, my plan is to use Flask to connect my back-end to a simple html/(possibly CSS) front-end. Ultimately, my goal is to build a web app that demonstrates the Scaffold parsing/identifying functionality in a clean and reliable way.

I am back and forth about when the appropriate time is to introduce a database component to my stack is. Keeping track of all of the documents that Scaffold is being fed as well as their results would be most appropriately accomplished by a database.

On the other hand, Scaffold does not need a database in order to perform its parsing and other token-tagging tasks for this iteration of the logic. For testing, I could stick with my plan to export Scaffold inputs and results to text documents and writing a Python program to parse those results for feedback figures.

What I really need to do is lay out what specific data I would like Scaffold to give me each time it runs, what metrics I want to pull out of that data, and what the best way to accomplish that is. Keeping in mind that I will almost certainly want to keep a record of the details of every instance that someone runs Scaffold. Okay. Yes. I almost certainly need a database, now that I think about it.

All in all, here's what I'm thinking for my current game plan -

->Build simple HTML front-end consisting of non-negotiable components
    ->Text box to collect input document
    ->'Go' Button to indicate that the contents of the input box should be processed by the back-end and made into a Scaffold
    ->area to display enumerated lines of original article
    ->tables for People/Locations/'Other Subjects' containing counts and line citations to specific occurrences
    ->some kind of minimally readable display of quotes/event dates and times/data points and their line numbers

->use Flask to render HTML templates and call back-end functionality (forgive me for mis-wordings, I literally started using Flask yesterday)
->Plan on a lot of debugging, rejoice if it is less than a lot
->Once the front-end can take in input, push it through the back-end meat grinder, and display the results, determine design of database I want to store inputs and results in
->set up MySQL database and add calls to store desired program data for each instance that Scaffold is run
->Plan on a lot of debugging, rejoice if it is less than a lot
->While on localhost, verify that there are no runtime errors for a basic set of test documents and edge cases
->verify that the database appears to be storing data as expected
-> Determine where I want to host a publicly accessible version of Scaffold and throw it up on there
->resume testing plans described in my first Scaffold blog post, modified as appropriate to incorporate the new set up

Hopefully, that turns out to be a reasonable course of action. If not, I'll rework it as necessary. For now, I just want to stop blogging about it and start building it!

Although I should probably make all of those bullet points into a todoist list first.