HELP! Convert text with database
Hey everyone I'm new to Rebol programming. I would like some help with my very first project. Maybe just point me in the right direction. There are a lot of tutorials on REBOL but there is so much that I have difficulty finding the right examples so I don’t know where to start.
This is the concept:
Part1
I want to write a program which searches on a specific website on synonyms and downloads all the words with their synonyms and writes them to a text file so like this:
Word 1
Synonym 1
Synonym 2
Word2
Synonym 1
Synonym 2
Synonym 3
Synonym 4
This is the website I want to use http://www.mijnwoordenboek.nl/synoniem.php . If you have a look at it you’ll see that each word has its own address http://www.mijnwoordenboek.nl/synoniemen/aan%20kaart like this and %20 represent a blank space. The program has to download all the words so it has to search every combination and I think 30 characters should do. Then it has to recognize which words represent the synonyms and save them too.
The first will be completed once this is done and I no longer will have use for the program because I will have the whole database in one big .txt file.
Part 2
I have to create a text field where the user can insert the text to convert it with a convert button. The program will search each of the words in the text in the database created and pick one of the synonyms randomly. This will be done real-time and if a word has more synonyms the user can select a certain word and using the up and down arrow keys on the keyboard change the synonym in the converted text in one of the other synonyms available in the database for the particular word. Eventually the user can save the newly created text as a .txt file on the desktop
Any suggestions, codes or links to tutorials which describe a certain part of this program or something similar that I can use are welcome. Thanks in advance
BTW: I would like to save this as an executable Windows program . I’ve read that this is possible but don’t know how.
posted by: wahagn 12-Apr-2014/11:11:26-7:00
For the first part this is all I have now. It's not working code but it's a start maybe some people can help me further and correct it.
R E B O L [title: "Text Converter part 1 the database"]
data: read http://www.mijnwoordenboek.nl/synoniemen/ + -------------------------------
(-----------------------) = every combination possible
while [
data: find text "van"
]
[
print [copy/part data (number of characters of words following)]
]
posted by: wahagn 12-Apr-2014/13:05:25-7:00
First program:
1) Create an empty block to start:
all-words: copy []
2) Use a loop to step through each of the pages A-Z
for letter #"A" #"Z" 1 []
3) Inside that loop, read each of the numbered pages 1 to last number, for each letter, and parse the word links on each page. The pages are in the format:
A
B
C
D
E
repeat num 500 [
; read page
; if the page doesn't exist, exit loop
]
Inside the loop, concatenate the number to the base URL, and parse:
words-on-page: copy []
page: read http://www.mijnwoordenboek.nl/engels/synoniemen/A/52.html
parse page [
any [
thru {href="http://www.mijnwoordenboek.nl/engels/synoniemen/} copy new-word to {">} (append words-on-page new-word)
]
to end
]
remove-each w words-on-page [find w ".html"]
editor words-on-page
3) Inside that loop, parse and collect the word links on each word page. The word links are in the same format:
href="http://www.mijnwoordenboek.nl/engels/synoniemen/azeri">azeri (3x)
azide (2x)
foreach w words-on-page [
; concatenate word with base URL
; same basic parse rules as above, pulling out each synonym link
; make a block of the parse synonyms, add the word and it's synonym block to 'all-words
]
Program 2:
After program 1, you'll have a block in the format:
all-words: [
"word1" ["synonym1" "synonym2" "synonym3"]
"word2" ["synonym1" "synonym2" "synonym3"]
]
You can use 'select to get the synonym block for any word:
select all-words "word1"
posted by: Nick 13-Apr-2014/8:58:02-7:00
Thank you.
Hmm I'm very new to REBOL but using the learn rebol page I had a crack at it. However I think I mixed some url functions with string related ones. This is the start I made:
R E B O L [title: "Text Converter Creat Database"] ; Set titel
all-words: copy [] ; Create a new block for all-words
words-on-page: copy [] ; Creat a new block for the words on the specfic page
page: http://www.mijnwoordenboek.nl/synoniemen/ ; Create variabele
for letter #"A" #"Z" 1 ; go form A to Z skipping by one
[
insert/dup end page letter ; Add the letter to the end of the page
while [ read page != 0 ] ; As log as the page exists do
[
repeat num 30 [
insert/dup end page "/" + num ; Add the number to the page
read page ; Read that page
]
]
]
posted by: Wahagn 13-Apr-2014/13:24:08-7:00
In your example you use a for loop with characters.
Same goes for this tutorial on learnrebol.com
REBOL will properly increment any data type that it understands: for alphabet #"a" #"z" 1 [print alphabet]
When I use this or the example you gave me I get an error saying: "Script error: For does not allow char! for its start argument"
What's wrong?
posted by: Wahagn 14-Apr-2014/12:46:34-7:00
Ah, you must be using R3. The 'for loop in R2 can use more datatypes than are currently supported in R3. You can use 'for or 'repeat to do this:
repeat i 26 [
char: pick "abcdefghijklmnopqrstuvwxyz" i
; print char
]
posted by: Nick 14-Apr-2014/15:01:05-7:00
Ok thanks. You're really a big help.
I got the first part down:
Program 1:
R E B O L [title: "Text Modifier: Database"] ; Set title
page: http://www.mijnwoordenboek.nl/synoniemen/A ; Create variable
all-words: copy [] ; Create a new block for all words
words-on-page: copy [] ; Creat a new block for the words on the specfic page
all-synonyms: copy[] ; Create a new block for all synonyms
synonyms-on-page: copy [] ; Creat a new block for the synonyms on the specfic page
for alphabet #"A" #"Z" 1
[
remove at tail page -1
insert at tail page 0 alphabet
print page
]
halt
Program 2:
R E B O L [title: "Text Modifier"]
do %r3-gui.r3
view [
text ""
text ""
text "© wahagn"
text ""
text ""
area "Paste the source in this textarea. Then press Change. A different text with the same meaning will be generated in the second textarea. It's possible to manually select a word in the output and use arrow keys to change it. Five languages are supported. Note that the best results are achieved when text is changed in little parts at one time."
hgroup[
radio "Dutch" radio "English" radio "German" radio "French" radio "Spanish"
return
button brown "Change"
button brown "Save"
]
area
text "http://wahagn.co.nf"
]
For the numbers in the url like /1 /2 etc. I have to use an integer and do something like count:1 and count: count + 1 but because it's an integer I have to turn it into a string for the url . I have not yet figured out how I can do this...
posted by: wahagn 14-Apr-2014/16:40:24-7:00
Here's a working solution:
R E B O L [title: "Synonym Viewer"]
if not exists? %synonyms.r [
request-download/to http://re-bol.com/synonyms.r %synonyms.r
]
synonyms: load %synonyms.r
view center-face layout [
f: field "word" [
if error? try [
a/text: copy {}
foreach w (select synonyms value) [
append a/text join w newline
]
show a
] [
a/text: copy { }
show a
]
]
text "Synonyms:"
a: area 500x500
]
I got the raw data from http://lingucomponent.openoffice.org/MyThes-1.zip and wrote this script to create the synonyms.r Rebol data file:
R E B O L []
print "working..."
synonyms: copy []
temp: copy []
dat: read/lines %th_en_US_new.dat
count: 1
foreach line dat [
count: count + 1
if 0 = mod count 10000 [print count]
either line/1 = #"(" [
words: next parse/all line "|"
append temp words
] [
append/only synonyms temp
append synonyms first parse/all line "|"
temp: copy []
]
]
print "Saving..."
save %synonyms.r at synonyms 4
; editor copy/part at synonyms 4 1000
halt
Using that file, all you need to do is:
select synonyms "yourword"
Although your first program is a nice parsing exercise, I think you're probably going to be shut down by the web site you're trying to scrape. Have you read their terms of use or license info? I suggest that it's better (and simpler, more practical, etc.) to use an existing data set which is freely available such as the one above.
posted by: Nick 14-Apr-2014/23:23:28-7:00
That is a very nice example. I can't run it yet because it gives me errors with center-face and if I delete that I get an error with view doesn't have a view-face argument.
Anyway I still want to finish the old version and test that one because I think the database was better in comparison. Especially because it supports 5 languages. Copyrights don't concern me at all because here in the Netherlands use of any copyrighted material for personal use is allowed and furthermore the database is not necessarily a part of the program. I can tell users to get their own "library" with the specified format.
If I succeed we can have two versions which is even better.
BTW: Your example is a synonym viewer. Note that my idea is to alter entire texts. By changing them by using synonyms.
posted by: Wahagn 15-Apr-2014/15:56:18-7:00
What sort of error are you receiving with center-face (again, this is R2 code, not R3). I'll give you some working code for the web site scraping when I get another moment. You'll need to extend the logic to alter each word in entire text. The simplest way would be to put each word in a text list. You could also parse out the word at the caret position in a text area and pop up a synonym selector.
posted by: Nick 15-Apr-2014/18:56:27-7:00
OK nice. I'll have a look at the error when I get home today. BTW I always test it with both versions to be sure but I save them as .r3 files.
Becuase I'm unfamiliar with Rebol it would be nice if you made a working code for parsing. But I think I can handle programming the logic part. I have some ideas on how to accomplish that and doing it yourself gives you experience.
posted by: Wahagn 16-Apr-2014/2:40:51-7:00
Ok I can run your code now and it's indeed a very nice synonym viewer. Although the synonyms are more sort of related words rather than synonyms but that's just the database;
I also coded a way to create a list of the url's for the website. I got the first part where it has to create url for letter A but after that I should use a little different code because you have to delete more characters from the string than the first time you run it. that is why I used an either statement:
Code:
count: 1
times: 0
for alphabet #"A" #"Z" 1
[
either(times < 1)
[
remove at tail page -1
insert at tail page 0 alphabet
insert at tail page 0 "/"
while [count <= 30]
[
either (count < 11)
[
insert at tail page 0 count
print page
remove at tail page -1
count: count + 1
]
[
remove at tail page -1
insert at tail page 0 count
print page
remove at tail page -1
count: count + 1
]
]
]
[
; When A is done the rest should be done this way
; no solution doun yet
]
count: 1
times: times + 1
]
List
http://www.mijnwoordenboek.nl/synoniemen/A/1
http://www.mijnwoordenboek.nl/synoniemen/A/2
http://www.mijnwoordenboek.nl/synoniemen/A/3
..etc till 30 ...
http://www.mijnwoordenboek.nl/synoniemen/A/28
http://www.mijnwoordenboek.nl/synoniemen/A/29
http://www.mijnwoordenboek.nl/synoniemen/A/30
Now the code for the rest of the alphabet :). and I should replace print by read in the end of course.
posted by: Wahagn 16-Apr-2014/14:28:50-7:00
I have made some more progress. Now I can print every letter in the alphabet with all page numbers.
R E B O L [title: "Text Modifier: Database"] ; Set title
page: http://www.mijnwoordenboek.nl/synoniemen/A ; Create variable
all-words: copy [] ; Create a new block for all words
words-on-page: copy [] ; Creat a new block for the words on the specfic page
all-synonyms: copy[] ; Create a new block for all synonyms
synonyms-on-page: copy [] ; Creat a new block for the synonyms on the specfic page
count: 1
times: 0
for alphabet #"A" #"Z" 1
[
if(times > 0)
[
repeat num 3
[
remove at tail page -1
]
]
either(times < 1)
[
remove at tail page -1
insert at tail page 0 alphabet
insert at tail page 0 "/"
while [count <= 30]
[
either (count < 11)
[
insert at tail page 0 count
print page
remove at tail page -1
count: count + 1
]
[
remove at tail page -1
insert at tail page 0 count
print page
remove at tail page -1
count: count + 1
]
]
]
[
insert at tail page 0 alphabet
insert at tail page 0 "/"
while [count <= 30]
[
either (count < 11)
[
insert at tail page 0 count
print page
remove at tail page -1
count: count + 1
]
[
remove at tail page -1
insert at tail page 0 count
print page
remove at tail page -1
count: count + 1
]
]
]
count: 1
times: times + 1
]
halt
posted by: Wahagn 16-Apr-2014/17:35:31-7:00
Nick, could you please help me with the parsing.
I put the first part in the code at the places where I had "print page" I put in the parsing example you made but all I get is an editor with just "[]"
So this is the whole code now:
R E B O L [title: "Text Modifier: Database"] ; Set title
page: http://www.mijnwoordenboek.nl/synoniemen/A ; Create variable
all-words: copy [] ; Create a new block for all words
words-on-page: copy [] ; Creat a new block for the words on the specfic page
all-synonyms: copy[] ; Create a new block for all synonyms
synonyms-on-page: copy [] ; Creat a new block for the synonyms on the specfic page
count: 1
times: 0
for alphabet #"A" #"Z" 1
[
if(times > 0)
[
repeat num 3
[
remove at tail page -1
]
]
either(times < 1)
[
remove at tail page -1
insert at tail page 0 alphabet
insert at tail page 0 "/"
while [count <= 30]
[
either (count < 11)
[
insert at tail page 0 count
parse page [
any [
thru {href=" http://www.mijnwoordenboek.nl/synoniemen/} copy new-word to {">} (append words-on-page new-word)
]
to end
]
remove-each w words-on-page [find w ".html"]
editor words-on-page
remove at tail page -1
count: count + 1
]
[
remove at tail page -1
insert at tail page 0 count
parse page [
any [
thru {href=" http://www.mijnwoordenboek.nl/synoniemen/} copy new-word to {">} (append words-on-page new-word)
]
to end
]
remove-each w words-on-page [find w ".html"]
editor words-on-page
remove at tail page -1
count: count + 1
]
]
]
[
insert at tail page 0 alphabet
insert at tail page 0 "/"
while [count <= 30]
[
either (count < 11)
[
insert at tail page 0 count
parse page [
any [
thru {href=" http://www.mijnwoordenboek.nl/synoniemen/} copy new-word to {">} (append words-on-page new-word)
]
to end
]
remove-each w words-on-page [find w ".html"]
editor words-on-page
remove at tail page -1
count: count + 1
]
[
remove at tail page -1
insert at tail page 0 count
parse page [
any [
thru {href=" http://www.mijnwoordenboek.nl/synoniemen/} copy new-word to {">} (append words-on-page new-word)
]
to end
]
remove-each w words-on-page [find w ".html"]
editor words-on-page
remove at tail page -1
count: count + 1
]
]
]
count: 1
times: times + 1
]
halt
posted by: Wahagn 19-Apr-2014/12:51:31-7:00
Ok I know now that you have to read the page first before parsing it. and instead of the editor I use print. But the only print I get is 1 paragraph of blank space so the parsing is still not working.
So this is the code now. The only thing you''ll see is that it prints a lot of blank space:
R E B O L [title: "Text Modifier: Database"] ; Set title
page: http://www.mijnwoordenboek.nl/synoniemen/A ; Create variable
all-words: copy [] ; Create a new block for all words
words-on-page: copy [] ; Creat a new block for the words on the specfic page
all-synonyms: copy[] ; Create a new block for all synonyms
synonyms-on-page: copy [] ; Creat a new block for the synonyms on the specfic page
count: 1
times: 0
for alphabet #"A" #"Z" 1
[
if(times > 0)
[
repeat num 3
[
remove at tail page -1
]
]
either(times < 1)
[
remove at tail page -1
insert at tail page 0 alphabet
insert at tail page 0 "/"
while [count <= 30]
[
either (count < 11)
[
insert at tail page 0 count
parse page [
any [
thru {href=" http://www.mijnwoordenboek.nl/synoniemen/} copy new-word to {">} (append words-on-page new-word)
]
to end
]
remove-each w words-on-page [find w ".html"]
print words-on-page
remove at tail page -1
count: count + 1
]
[
remove at tail page -1
insert at tail page 0 count
parse page [
any [
thru {href=" http://www.mijnwoordenboek.nl/synoniemen/} copy new-word to {">} (append words-on-page new-word)
]
to end
]
remove-each w words-on-page [find w ".html"]
print words-on-page
remove at tail page -1
count: count + 1
]
]
]
[
insert at tail page 0 alphabet
insert at tail page 0 "/"
while [count <= 30]
[
either (count < 11)
[
insert at tail page 0 count
parse page [
any [
thru {href=" http://www.mijnwoordenboek.nl/synoniemen/} copy new-word to {">} (append words-on-page new-word)
]
to end
]
remove-each w words-on-page [find w ".html"]
print words-on-page
remove at tail page -1
count: count + 1
]
[
remove at tail page -1
insert at tail page 0 count
parse page [
any [
thru {href=" http://www.mijnwoordenboek.nl/synoniemen/} copy new-word to {">} (append words-on-page new-word)
]
to end
]
remove-each w words-on-page [find w ".html"]
print words-on-page
remove at tail page -1
count: count + 1
]
]
]
count: 1
times: times + 1
]
halt
posted by: Wahagn 24-Apr-2014/15:31:35-7:00
Wahagn,
It seems like you're trying to do manually what the loops and concatenation accomplish for you. My first post explained everything you need - here's a full working script based on that explanation:
http://re-bol.com/mijnwoordenboek-synonymns.r
I set the repeat number to 3, which will get the first 3 pages of every letter. You can choose how to handle the logic. The simplest solution would be to set the repeat number to some number greater than the larger possible number of pages for any given letter, and break out of the loop when a read error occurs. You could also manually collect a list of the number of pages on their site for each letter, save each of those numbers in a block and loop the exact number of pages counted for each letter.
Either way, if you want to try doing something like this in production, you should add some more error checks, handle timeouts, the potential that their servers will stop serving you for scraping every page of their site, etc. (keep a list of the last pages, words, etc. downloaded, so you can resume interrupted sessions). And be aware that downloading each word page appears to take a good fraction of a second (from their servers - nothing to do with Rebol), so you're likely looking at very least 24 hours of constant downloading just to get your word list (if they don't blacklist you first). I'd still recommended using any freely available synonym database, instead of copying their content.
Hopefully the general technique here is useful :)
posted by: Nick 26-Apr-2014/0:48:46-7:00
Oh yes, i see now that it could be done easier and shorter. Indeed it's very useful to me so thank you.
Hmm.. maybe I will check for errors manually. For example I could add a save file line at the end of the for loop. This way I can make folders for every language and sub folders for every letter in the alphabet. Every time the letter is ready it would save it as A.txt, B.txt . I can look at the file and see if there are any errors then I will move on to the next letter.
Dutch: A.txt, B.txt, C.txt .. etc.
English: A.txt ... etc
Spanish: A.txt .. etc
...
...
etc.
The real applet which will do the text modifying will look at each word and see what the first letter is and only search in the specific text file of this letter to find that word and replace it by an appropriate synonym. So it doesn't have to scan the whole database with every word it needs to change.
By the way what is bugging me is that when reading the words it doesn't recognize special characters which are used in the other languages. How can this be solved?
posted by: Wahagn 27-Apr-2014/7:46:56-7:00
I'm trying to save the words and synonyms. The program has to recognize each of the different synonyms and some of them consist of more words so they should be written separated. In your example they are separated like this ["synonym", "synonym2" ] Which is fine actually but when I write them to a file they aren't and nore are they written on a new line. I don't know where to place new-line/ and how to separate the by commas.
This what I tried:
print uppercase w
probe synonyms-on-page
write/append/lines %A.txt w
write/append/lines %A.txt form synonyms-on-page
But the result is like this
WORDSYNOMYM SYNONYM2 SYNONYM3
instead of
WORD
SYNONYM, SYNONYM2, SYNONYM3
So how can I change the lay-out? Where should I place new-line/ and how separate them by commas??
posted by: Wahagn 30-Apr-2014/8:45:28-7:00
Those 'print and 'probe lines were there just to give you some feedback that the items were being parsed.
You should save them to a block. For example, before the foreach loop:
temp-block: copy []
Then in place of the print and probe lines (or in addition):
append temp-block uppercase w
append/only temp-block synonyms-on-page
You can save the block to a file with the 'save function:
save %filename temp-block
If you want to append that form of data to an existing file:
write/append %filename mold/only temp-data
posted by: Nick 30-Apr-2014/17:28:57-7:00
Ah ok I got that but how do I save them on different lines because append/lines and save/lines don't work. And without that I get one very big string. When changing words I need to find that word in the word-list and if it's one big string I might find it earlier on in one of the synonyms.
So it would be better to have
WORD
synonym, synonym, synonym
And the use a code like this:
read filname.txt
skip a line after every read line
to only search in the words for the specific word I want to change.
Of course I also could separate the words and synonyms in two .txt files where the number of the line for the word corresponds with the number of the line with the synonyms. Which approach is better in your opinion? Anyway I still have to save them on different lines for this way also.
posted by: Wahagn 2-May-2014/7:45:30-7:00
Oh I see now that the words and synonyms were saved in the same way in your version and that works fine.
So it was already right. I will let the program run and get the English synonyms.
But I still haven't found any solution for the other languages that have special characters that the program can't read. So any thoughts on that??
BTW: The downloading is really fast I think it just takes about 4 hours for the English language instead of the 24 you suggested
posted by: Wahagn 3-May-2014/10:31:58-7:00
Ok I still want to finish this prog. I try to use the same method you did in your example in my final graphical application. But there is a problem my application doesn't run in r2 and your code doesn't run in r3 so when I paste your code in my code and edit it I can't run the application anymore
Anyway this is what I tried and quotes for what I want to achieve (written for r3) :
R E B O L [title: "Text Modifier"]
do %r3-gui.r3
synonyms: load %English.txt
view [
text ""
text ""
text "© Wahagn"
text ""
text ""
i: area "Paste the source in this textarea. Then press Submit and Change. A different text with the same meaning will be generated in the second textarea. It's possible to manually select a marked word in the output and change it. English is supported by our library for other languages make your own. Note that the best results are achieved when text is changed in little parts at one time."
hgroup[
radio "Dutch" radio "English" radio "German" radio "French" radio "Spanish"
return
button red "Submit" on-action
[
;for each word that corresponds with one of our library words
;mark it ( make bold)
o/text: copy {}
foreach w (select synonyms value) [
append a/text join w newline ]
show o
]
button red "Change"
on-action
[
;for each word that is marked get the synonyms
;save all synonyms so user can still switch between them
;but choose one at random to replace the specific words
]
button black "Save"
on-action
[
;Get the text as a string with the chosen synonyms from the lists
;Save as .txt file
]
]
o: area
]
posted by: Wahagn 21-May-2014/14:07:53-7:00
Okay I removed all the extra graphic features to
make it easier to test. I got the first part working. Which is getting the text from one area and pasting it in the other:
R E B O L [title: "Text Modifier"]
do %r3-gui.r3
synonyms: load %English.txt
text: copy []
view[
i: area 300x100
button "Submit" on-action
[
append text copy get-face i
set-face o text1
;for each word that corresponds with one of our library words
;mark it ( make bold)
]
o: area 300x100
]
I think this is enough for tonight I'll try to make the thinks after the ";" tomorrow. Anyway I have the start
posted by: Wahagn 23-May-2014/17:40:38-7:00
Ok I'm a little bit further
R E B O L [title: "Text Modifier"]
do %r3-gui.r3
synonyms: load %English.txt
view[
i: area 300x100
button "Submit" on-action
[
text: copy []
append text copy get-face i
changingtext: copy[]
foreach w text ;find all the corresponding synonyms save them
;
set-face o text
]
o: area 300x100
]
posted by: Wahagn 24-May-2014/7:27:21-7:00
I haven't figured it out yet but I think I'm on the right track with this. Am I?
R E B O L [title: "Text Modifier"]
do %r3-gui.r3
synonyms: load %English.txt
view[
i: area 300x100
button "Submit" on-action
[
text: copy []
append text copy get-face i
foreach w text [ ; for each word in text
if find w synonyms [ ; if synonyms are found
join o/text textlist select syonyms value ]] ; replace the word with a textlist with the values of the synonyms
set-face o text
]
o: area 300x100
]
posted by: Wahagn 26-May-2014/16:21:33-7:00
|