Ph.D. Defense

Multi-Source Option-Based Policy Transfer

James MacGlashan

10:00am Friday, 25 January 2013, ITE 325B

Reinforcement learning algorithms are very effective at learning policies (mappings from states to actions) for specific well defined tasks, thereby allowing an agent to learn how to behave without extensive deliberation.  However, if an agent must complete a novel variant of a task that is similar to, but not exactly the same as, a previous version for which it has already learned a policy, learning must begin anew and there is no benefit to having previously learned anything. To address this challenge, I introduce novel approaches for policy transfer. Policy transfer allows the agent to follow the policy of a previously solved, but different, task (called a source task) while it is learning a new task (called a target task). Specifically, I introduce option-based policy transfer (OPT). OPT enables policy transfer by encapsulating the policy for a source task in an option (Sutton, Precup, & Singh 1999), which allows the agent to treat the policy of a source task as if it were a primitive action. A significant advantage of this approach is that if there are multiple source tasks, an option can be created for each of them, thereby enabling the agent to transfer knowledge from multiple sources and to combine their knowledge in useful ways. Moreover, this approach allows the agent to learn in which states of the world each source task is most applicable. OPT's approach to constructing and learning with options that represent source tasks allows OPT to greatly outperform existing policy transfer approaches. Additionally, OPT can utilize source tasks that other forms of transfer learning for reinforcement learning cannot.

Challenges for policy transfer include identifying sets of source tasks that would be useful for a target task and providing mappings between the state and action spaces of source and target tasks. That is, it may not be useful to transfer from all previously solved source tasks. If a source task has a different state or action space than the target task, then a mapping between these spaces must be provided. To address these challenges, I introduce object-oriented OPT (OO-OPT), which leverages object-oriented MDP (OO-MDP) (Diuk, Cohen, & Littman 2008) state representations to automatically detect related tasks and redundant source tasks, and to provide multiple useful state and action space mappings between tasks. I also introduce methods to adapt value function approximation techniques (which are useful when the state space of a task is very large or continuous) to the unique state representation of OO-MDPs.

Committee: Dr. Marie desJardins (Chair), Dr. Tim Finin, Dr. Michael Littman, Dr. Tim Oates, Dr. Yun Peng

Photo: ua-net.eu

The ESNET Lab in UMBC's Computer Science and Electrical Engineering department is looking for qualified undergraduate or graduate students to join and work on an exciting underwater sensor networks project. For more information go to http://esnet.cs.umbc.edu/eduwan.html.

The candidates applying should have experience in (or at least familiar with) some of the following areas:

1. FPGA development using Verilog/VHDL
2. Data Frame synchronization for wireless (or acoustic) communication
3. Doppler resistant compensation for FSK modulated data
4. Digital Signal Processing
   a) Adaptive data equalization (LMS/NLMS)
   b) Error Correction coding on FPGA, i.e. turbo codes, LPDC codes.

To apply please e-mail your resume and a cover letter detailing experience in the above mentioned areas to: Sorry, you need javascript to view this email address.

Photo: ua-net.eu

The ESNET Lab in UMBC's Computer Science and Electrical Engineering department is looking for qualified undergraduate or graduate students to join and work on an exciting underwater sensor networks project. For more information go to http://esnet.cs.umbc.edu/eduwan.html.

The candidates applying should have experience in (or at least familiar with) some of the following areas:

1. FPGA development using Verilog/VHDL
2. Data Frame synchronization for wireless (or acoustic) communication
3. Doppler resistant compensation for FSK modulated data
4. Digital Signal Processing
   a) Adaptive data equalization (LMS/NLMS)
   b) Error Correction coding on FPGA, i.e. turbo codes, LPDC codes.

To apply please e-mail your resume and a cover letter detailing experience in the above mentioned areas to: Sorry, you need javascript to view this email address.

The Center for Women in Technology (CWIT) welcomes Cindy Greenwood as their new Assistant Director. Ms. Greenwood will spearhead the new Cyber Scholars Program which kicks off this fall.

Cindy Greenwood loved college so much that she never wanted to leave.

So she didn’t.  

When an Advertising and Public Relations internship during her senior year of college showed Ms. Greenwood that it wasn’t the career for her, she switched gears. She let her heart decide.

“I felt like I could make more of a difference by going into higher education,” she says.

Greenwood knows first-hand the difference that the college experience can make. Raised in Ishpeming, Michigan–a small town of no more than 7,000 people that is “half the size of UMBC,” she says—Ms. Greenwood thought it would always be her home. That is, until she left for Grand Valley State University.  

“A college campus is like no place else. You can do anything,” says Ms. Greenwood. “You can go from a cultural event where you’re trying food from Cambodia, to a dance party with glowsticks.” The atmosphere of possibility urged her to try new things, like studying abroad in Australia.

Afterwards, a master’s program in Higher Education Administration at the Leadership Center of Washington State University beckoned to her. After graduating, Ms. Greenwood spent eight years working for and with college students, first at Ferris State University in Michigan, and then at the University of South Florida.

In 2011, she joined UMBC as the Alumni Programming Coordinator in the Office of Institutional Advancement. Here, she started the Student Alumni Association to help connect current students with alumni. Hungry for more one-on-one time with undergraduates, Ms. Greenwood volunteered to be the advisor for the UMBC Vegetarian Student Group.

It’s the chance to work with students on a daily basis that drew her to the Assistant Director position in the Center for Women in Technology, she says. Ms. Greenwood will coordinate the new Cyber Scholars Program, which is run in partnership by CWIT and the UMBC Center for Cybersecurity. Her duties include overseeing the Cyber Scholars Living Learning Community, planning events, advising scholars, and teaching a seminar and bridge program for Cyber Scholars.

“The scholars programs are really interesting [at UMBC] because they really touch on every part of students’ lives,” says Greenwood.

An advocate of social justice and equality, Ms. Greenwood says she identifies with CWIT’s mission to bolster support for women in the male-centered fields of Engineering and Information Technology. She is a co-chair of UMBC’s Presidents Commission of Women. In the end it all comes back to her experience in college.

“I’ve had some good female mentors throughout my career,” says Ms. Greenwood, “and I definitely hope to be that to other females as well.”

*Ms. Greenwood joins CWIT as Assistant Director on January 28, 2013.

It’s not too late to apply for the 2013 CRA-W Graduate Cohort Workshop scheduled for April 5-6, 2013 in Boston, MA. This event brings together women graduate students in their first three years of graduate school for a series of presentations and panels with successful senior women researchers from academic, industrial, and government laboratories about how to succeed in graduate school and in a research career.

Applications will be accepted until 11:59 p.m. (ET) today, 15 January 2013 via an online form. Applicants must be female students in their first, second, or third year of graduate school in computer science and computer engineering or a closely related field at a U.S. or Canadian institution. Past workshops provided support for travel expenses, meals, and lodging for students chosen to participate in this program and we anticipate that similiar support will be available in 2013.

It’s not too late to apply for the 2013 CRA-W Graduate Cohort Workshop scheduled for April 5-6, 2013 in Boston, MA. This event brings together women graduate students in their first three years of graduate school for a series of presentations and panels with successful senior women researchers from academic, industrial, and government laboratories about how to succeed in graduate school and in a research career.

Applications will be accepted until 11:59 p.m. (ET) today, 15 January 2013 via an online form. Applicants must be female students in their first, second, or third year of graduate school in computer science and computer engineering or a closely related field at a U.S. or Canadian institution. Past workshops provided support for travel expenses, meals, and lodging for students chosen to participate in this program and we anticipate that similiar support will be available in 2013.

It’s not too late to apply for the 2013 CRA-W Graduate Cohort Workshop scheduled for April 5-6, 2013 in Boston, MA. This event brings together women graduate students in their first three years of graduate school for a series of presentations and panels with successful senior women researchers from academic, industrial, and government laboratories about how to succeed in graduate school and in a research career.

Applications will be accepted until 11:59 p.m. (ET) today, 15 January 2013 via an online form. Applicants must be female students in their first, second, or third year of graduate school in computer science and computer engineering or a closely related field at a U.S. or Canadian institution. Past workshops provided support for travel expenses, meals, and lodging for students chosen to participate in this program and we anticipate that similiar support will be available in 2013.

TL;DR version: if you have Java 7 on your computer, update it. Now.

The Java programming language helped to make the Web popular in the mid 1990s by its invention of the Web applet. The idea was powerful and enabled complex programs to be automatically downloaded when you visited a Web page and run securely on your personal computer in a sandbox. The sandbox prevented any applet running in it from doing the things on your computer you would not want it to do — like accessing your files, sending email, accessing other computers on the Internet, or installing new programs.

If you've read any technology news in the past week you know that several very nasty security vulnerabilities were discovered in Java version 7 that can be exploited by attackers, allowing them to create applets that can execute arbitrary code on your computer.

The exploit applies to Java 7 and not to earlier versions of Java and only when that is used to run a Java applet in your browser. The problem does not affect the use of Java 7 on servers, in Java desktop applications, or embedded Java.

Today Oracle released a security alert to address the vulnerabilities. The vulnerabilities are remotely exploitable without authentication and have the highest CVSS severity score of 10. Oracle recommends that their fix be applied as soon as possible — some exploits are already available in hacking tools like Metasploit, which means that relatively unsophisticated people can use them.

Java developers and programmers should download the latest Java SE JDK and JRE 7 releases from Oracle and users running Java SE with a browser should download the latest JRE 7 from java.com. Windows users can also use automatic updates to get the latest JRE 7 release.

You can test the version of Java available to your browser <a href="http://www.java.com/en/download/installed.jsp">here</a&gt;.  Note that Google's popular Chrome browser does not support Java 7, so Chome users need not update Java unless they also use another browser, such as Firefox or Safai.

Oracle also changed the default security setting for Java to "High," which means that most users will need to approve Java applets before they run.

Making Sense

For his master's thesis, Computer Engineering student Kirit Chatterjee worked with scientists from UMBC's Center for Advanced Sensor Technology (CAST) on an innovative temperature sensor for premature babies.

In hospitals, doctors use a thermistor probe to monitor the temperature of a premature baby. But, the glue used to attach it is harmful for the baby, whose skin is as fragile as tissue paper.

“When the probe is removed, there is a high risk of “epidermal stripping” occurring-i.e. the skin of the baby can tear, leaving it open to infection,” explains Kirit Chatterjee, a Computer Engineering graduate student. For his master’s thesis, Kirit helped develop a new temperature sensing device that avoids this problem.

It wasn’t easy. During research, other obvious options had been shot down one by one: Bluetooth sensors had batteries that leaked toxic chemicals. Wireless sensing devices emitted energy that was harmful for the baby.

The solution, supported by an NIH grant, and later commissioned by General Electric (GE), was the result of the combined brain-power of a group of UMBC scientists led by Dr. Govind Rao, Director of UMBC’s Center for Advanced Sensor Technology (CAST). Dr. Yordan Kostov (CAST), Dr. Hung Lam (CAST), and Dr. Ryan Robucci (CSEE)–Kirit’s advisor for the project–were the team’s key players.

Together they created a patch containing a unique fluorescent orange dye. “The intensity of the orange emission depends on the temperature,” explains Dr. Kostov, the senior scientist on the temperature project who also worked closely with Kirit. When the baby’s temperature rises, he explains, the orange patch becomes brighter.

Recording and translating the patch’s fluorescence into a temperature reading that a doctor could understand was Kirit's job. “My part,” explains Kirit, “was to take care of the Engineering side—namely, to build the sensing apparatus.”

Choosing the right camera to monitor the dye was another challenge. The project was bound to a strict budget since the new sensor system was slated for mass production by GE. Therefore, expensive scientific cameras were out of the question.

Instead, Kirit reverse engineered and manipulated a much more affordable camera to serve his purposes. He used a two megapixel camera–the same camera found inside an iPhone 3–to monitor the dye in the patch.

“The dye is just the target for the Computer Engineer,” he says. “To the engineer, it’s just photons being emitted which translate to analog voltage signals inside the camera which then translate to digital bits inside the FPGA and then are analyzed.”

Next, Kirit used an FPGA in order to tap into the camera and retrieve its data, and MATLAB to translate the data into a traditional temperature reading.

The result is a temperature sensing device that is affordable, accurate, and, most importantly, safe for the baby. Dr. Kostov explains that when the GE contract comes to an end this September, the patch system will undergo clinical trials and toxicology tests. If all goes well, the new system should be found in premature baby incubators across the world in as little as two years.

Congratulations to our December Ph.D. graduates! Read on to hear about their Ph.D. dissertation research and their plans for the future. 

	Dr. Karuna Joshi Computer Science Semantically Rich, Policy Based Framework to Automate Lifecycle of Cloud Based Services Mentors: Yelena Yesha and Tim Finin Thesis Topic: Dr. Joshi developed a new framework to automate the acquisition, composition, and consumption/monitoring of virtualized services delivered on the cloud. The lifecycle consists of five phases of requirements, discovery, negotiation, composition, and consumption. She has developed ontologies to represent the concepts and relationships for each phase using Semantic Web languages. She has also developed a protocol to automate the negotiation process when acquiring virtualized services. "I chose to concentrate on Cloud Services automation for my Ph.D. thesis since I was able to draw upon my extensive experience as an IT Project Manager to determine open issues that need to be addressed for broader adoption of cloud computing." Future plans: Dr. Joshi has received funding from NIST to continue her research on Cloud Computing and Big Data management. As part of this funding, she will be working as a research faculty member in the CSEE Department. In the spring, Dr. Joshi will teach a course on Software Design and Development.
	Dr. Phuong Nguyen Computer Science Data Intensive Scientific Compute Model For Multicore Clusters Mentors: Milton Halem and Yelena Yesha Thesis Topic: Dr. Phuong developed a scalable workflow system on top Apache Hadoop for orchestrating data intensive scientific workflows. New scheduling algorithms have been developed in the workflow system to manage and reduce latency of the workflow executions. The evaluations of the workflow system on the climate data processing and analysis application (several TB dataset) showed that it is feasible and improved. The scientific results of the application provide new global climate change indicators for the decade of 2002-2012. "The Ph.D. topic came from the motivations related to our NASA and NOAA projects which need to process and analyze very large datasets to study climate change. My research contributions provide new tools for accelerating scientific discoveries from very large datasets and the scientific results." Future plans: Work on research and development related to building large distributed systems or applications.
	Dr. David Chapman Computer Science A Decadal Gridded Hyperspectral Infrared Record for Climate Mentors: Milton Halem Yelena Yesha, Shujia Zhou, John Dorband, Joel Susskind (NASA) Thesis Topic: Dr. Chapman helped improve our understanding of Global Climate Change by creating a Climate Data Record (CDR) of Outgoing Longwave Radiation (OLR) from 55 terabytes of NASA satellite weather observations from the Atmospheric Infrared Sounder (AIRS). He developed a parallel data-intensive scientific workflow infrastructure making use of Large Array Storage (LAS) in order to show the complete derivation these climate trending results. "Global Climate Change and Global Warming are very important and controversial issues, and we need to measure if they have actually happened. AIRS is the first of its kind because it measures hyperspectral radiation. The trick is to take a Big Dataset, and squeeze it into something meaningful. This takes a lot of hardware, and typically a large software team to develop the processing system. I showed how the Large Array Storage (LAS) paradigm can simplify these calculations along with their derivation." Future plans: Dr. Chapman has applied for a post doc in Climate Modeling at Columbia University. It would allow him to do interdisciplinary work to develop Big Data Analytics infrastructure alongside the statistical validation of climate models.
	Dr. Niyati Chhaya Computer Science Joint Inference for Extracting Soft Biometric Text Descriptors from Patient Triage Images Mentors: Tim Oates Thesis Topic: Dr. Chhaya's research was a combination of Soft biometrics, Probalistic Graphical Models, and Natural Language Processing techniques. The aim was to extract soft biometric text labels (using computer vision techniques) from images of mass disaster victims. The main contributions of the work include soft biometric feature extractors, a probalistic graphical model that exploits related appearance-related features, and a novel study of natural human descriptors using NLP techniques that help understand 1) how people describe other people and 2) order and structure of free text human descriptions. "Socially, this work aims at addressing the issue of providing victim information to the public in a post disaster situation. It forms an important contribution to anonymize available image data using text labels to facilitate efficient search. Technically, this is the first work of its kind that aims at using Probabilistic Graphical Models to relate Soft biometric features, and in turn improve the overall accuracy of text label extraction. Also, the NLP study is a significant contribution along with the datasets gathered for this research. The key contribution is the use of techniques from computer vision, machine learning, and NLP to build a robust system that extracts soft biometric features." Future Plans: Dr. Chhaya has moved back to India and will work as a Computer Scientist with Adobe Research Labs starting in January.
	Dr. Yasaman Haghpanah Jahromi Computer Science A Trust and Reputation Mechanism Through Behavioral Modeling of Reviewers Mentors: Marie desJardins Thesis Topic: Dr. Haghpanah introduced a novel mechanism to represent trust and reputation using behavioral modeling of online reviewers. Her approach helps decision makers utilize reputation information more effectively. "Evidence shows that people are now relying more and more on other people's posted opinions for making decisions about which product to buy, which movie to watch, etc. So, I modeled the raters' or in general information providers' behavior and showed how we can improve our decisions by knowing the behavior of the online raters." Future Plans: Dr. Haghpanah is currently interviewing for postdoctoral positions at universities and research labs to extend and broaden her knowledge.
	Dr. Ganesh Saiprasad Electrical Engineering Automatic Detection of Adrenal Gland Abnormality Using The Random Forest Classification Framework combined with Histogram Analysis Mentors: Chein-I Chang Thesis Topic: Dr. Saiprasad proposed a new, more accurate way to detect adrenal abnormalities: rather than using the popular Region of Interest (ROI) method, Dr. Saiprasad suggests segmenting the adrenal gland automatically using the random forest classification framework and then performing histogram analysis. "Working with radiologists and surgeons at the University of Maryland Medical Center on my Master's research helped me pick a topic for my Ph.D. research. Adrenal gland abnormality detection is a very challenging problem and we have some preliminary results now to show that it can be done automatically. This is a very important step forward in using such systems as decision support tools and also the same methodology can be used for other smaller organs to detect abnormalities which are challenging to detect on CT." Future Plans: Postdoc at the National Institute of Standards and Technology (NIST)
	Dr. Kevin Fisher Computer Engineering Real-Time Progressive Band Processing for Linear Spectral Unmixing and Endmember Extraction Mentors: Chein-I Chang Milton Halem (NASA) Thesis Topic: Dr. Fisher developed three algorithms that work on hyperspectral images–pictures (often taken by satellites or airborne cameras) where each pixel is a spectograph of the materials in that part of the image. His algorithms work to reduce the amount of irrelevant data in the image, detect samples of pure materials in the image, and then estimate the abundance of those materials in each pixel in the image. "In 2006, I finished a Master's thesis with Prof. Alan Sherman on electronic voting systems. It was an engaging project in a hot topic in computing, but it was not related to the work I was doing as an intern at NASA Goddard Space Flight Center. I sat down with my supervisor and some NASA technologists, and looked for common areas of interest between UMBC and NASA. Hyperspectral image processing was on the short list and that's when I contacted Prof. Chein-I Chang about potential research projects." Future Plans: Dr. Fisher will continue working at NASA as a software systems engineer working on the ground antenna system for the Geostationary Operational Environmental Satellite, R-Series (GOES-R) spacecraft, a new line of weather satellites due to launch in 2015.
	Dr. Joel Sachs Computer Science Supporting Citizen Science and Biodiversity Informatics on the Semantic Web Mentors: Tim Finin Thesis Topic: Dr. Sachs introduces an approach to constructing ontologies by layer, designed to make it easier for both data publishers and application developers to tailor-fit semantics to use cases.